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PREFACE 



This booklet has been prepared as a guide to evaluation 
for educational decision-makers. It is intended primarily 
for administrators although it may be appropriate to other 
decision-makers as well, e.g., School Board members and 
teachers. The major purpose of this booklet has been to 
discuss the ''what'' and "why'' of evaluation rather than the 
"how to." 

After dealing with some basic concepts of evaluation 
to clarify misunderstandings and misinformation. Dr. Benedict 
deals with the practical steps of evaluation: Who should 
negotiate the contract? Who initiates evaluation? What 
are "goals process" and "parts process" and how are they 
matched? What are the steps in putting the process of 
evaluation into operation? What are the criteria for 
assessing observational techniques? What will a decision- 
maker do with an evaluation report? What can the decision- 
maker do if a school district has limited resources? 

Hven though Dr. Benedict has avoided as much "jargon" 
as possible, some new terminology has been necessary. 
Terms such as prioratize or operationalization may be 
"fuzzy" or have a different connotation than might or- 
dinarily be attached to them. Therefore a glossary has 
been provided at the end of the booklet. 

In introducing this guide. Project Evaluation, Capitol 
Region Education Council consi-^'^^rs it a step on the long 
road towards effective evaluation. I hope that it will be 
widely used among decision-makers for the betterment of the 
educational process. 

Philip S. Saif, Director 
Project Evaluation 



INTRODUCTION 



The purpose of this booklet is to preseht an intro- 
duction to educational evaluation for certain educational 
decision-makers, i.e.. administrators. It has not tried to 
present a step by step of "how to do^' evr^luation but rather 
has concentrated on presenting basic concepts of evaluation. 

This booklet also does not try to present one "model'* 
of evaluation. This is clearly beyond the scope of this 
work. There are several models of educational evaluation 
currently available in education. An administrator or any 
decision-maker would probably wapt to consider each model, 
its advantages and disadvantages, before making a decision 
to choose one or another of then. 

It is sufficient to refer to these models in an intro- 
duction. The most notable model of evaluation is that or- 
iginated by Stufflebeam and others working with him. This 
model is called the CIPP evaluation model and actually pro- 
vides for four different phases or types of evaluation: 
Context, Input, Process and Product. Of the four or five 
leading models, this one is perhaps the most detailed. Its 
major purpose is to provide data to educational decision- 
makers to help them in making decisions. 

The second most detailed model is probably that of 
Provus and is called the Discrepancy Model. As with the 
CIPP model, the purpose of this model is to provide data 
to decision-makers for their decision making needs. 

The Fortune/Hutchinson evaluation methodology is some- 
what different than the models in that while it too is an 
approach to evaluation, its purpose is to provide data to 
educational decision-makers. It is much more detailed and 



has a more specific set of rules and procedures than do the 
dther two models just described. Both the CIPP and the Dis- 
crepancy Models are somewhat less operational than is the 
F/H at this point in time. 

Aside from these three, the ether "models'* which exist 
are really just conceptual moUels. That is, at this point 
in time, they have not really been developed sufficiently 
to allow for a full scale evaluation to be done using them. 
They do not contain sufficient detail or methodology for 
implementing them on the local level. 

One of these i£ Stake's Countenance Model, and the 
other is Scriven's Goal-Free Model. Both of these are pii- 
marily conceptual models of evaluation. Both provide for 
a very high degree of judgment on the part of the evaluator 
about the project or enterprise being evaluated. Popham and 
Baker have written several works on the evaluation of in- 
struction. The major thrust of these works has been the 
evaluation of instructional objectives. The major thrust 
of the others mentioned above, would seem to be on a lar- 
ger or broader scale including the evaluation of objectives 
but not limited to them. 

Detailed information about these models can be found 
at the CREC Evaluation Center and the reader is referred 
to that resource center for additional help and information 
about evaluation models. The reference section provided 
at the end of this booklet also provides a rather detailed 
list of evaluation materials currently available in this 
field. 

There is one final point to note about this booklet. 
The format presented takes the perspective that the eval- 
uator is an external person with whom an administrator or 
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decision-maker would contract for evaluation services. The 
point of view has been taken deliberately for illustrative 
purposes. It does not mean that the evaluator has to or 
should be an ^'external" person. 

In education, teachers and administrators are often 
required to wear many hats or to play many roles. The 
teacher is both instructor and evaluator of classroom 
learning. The superintendent or principal is both admin- 
istrator and evaluator of his personnel. In other words, 
the same person may at different times play the role of 
evaluator as well as another role. Trying to present 
evaluation from this perspective would have been confus- 
ing and difficult to comprehend. Therefore, it was de- 
cided to separate these roles in this booklet and present 
the material as if the evaluator were an external person. 

However, this booklet can be read as though the eval- 
uator was an internal person or even as though the evalu- 
ator was one role which an administrator might play. This 
would require a few mechanical differences • For example, 
in the section dealing with the negotiation of the contract 
the signing of the "contract** might be omitted, if the 
evaluator and administrator were one person. However, 
the information needed in such a contract would still be 
detailed and still be needed. It might not be called a 
**con tract'* or be a formal document, but the information 
would still be needed. 

It has not been the intention oi this booklet to in- 
fer that evaluation can only be done by external evaluators 
In fact, evaluation is probably more often done by internal 
rather than external persons. It has not been the inten- 
tion of this booklet to present one model over another but 
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rather to present some questions and concerns which might 
be hold regardless of the model used. Finally, it has not 
been the intention of this work to present a guide for all 
educational decision-makers from elementary teachers to 
college teachers. The primary point of view taken has 
been for administrators. Much of the material herein will 
probably be useful to them; that is fine and it is hoped 
that other decision-makers might use it. The main purpose 
of this booklet has been to present some introductory 
material on educational evaluation to educational admin* 
istrators on the elementary and secondary school level. 



Larry G. Benedict, Ed.D, 
August 1973 
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I. AN INTRODUCTION TO EDUCATIONAL EVALUATION 



The starting point in evaluation occurs well before 
the evaluation begins. That point should be when one asks, 
and answers, the question: "Wh{/ do I ivant to evaluate?" 
Unless this question is answered, an evaluation should not 
be undertaken because, in fact, maybe it is not evaluation 
that is needed or wanted, but something else. 

Here are some typical reasons for wanting to have an 
evaluation : 

(1) For public relations so someone will like me, 
or fund me, etc, 

(2) To find out what the students need. 

(3) To make program or planning decisions, 

(4) To provide systematic, ongoing information (data) 
as a basis for making decisions. 

However, noc all of these ^.i:* evaluation, so a decision 
maker would not (should not) hire an evaluator to do all of 
these. For example, evaluation is fundamentally different 
from a public relations (PR) job. PR brings to mind Madison 
Avenue, marketing, public image and so on. This is not to 
say that a PR man might not want to avail himself of some of 
the data an evaluation design would collect. This is to say 
however, that the evaluation designer's job is not PR* If 
an enterprise wishes to sell itself to the public, it hires 
a PR expert, goes to an advertising agency or buys commer- 
cial time. If an enterprise desires objective, systematic 
feedback about the status of that enterprise, it hires an 
evaluator or evaluation designer. 



It is Important not to confuse the roles of PR and 
evaluation, for the methods, nature find goals of each are 
fundamentally different. A PR expert is in a position to 
do a much better job of promoting one's image or selling 
one's wares than is a person trained only in evaluation. 
Conversely, a PR man is not usually equipped or skilled 
iii evaluation design. Basically, then, this simple rule 
of thumb should be remembered: If one wants a PR job, 
kyLKd a PR man; if one wants an evaluation design, 
an tvatuato/i. 

The same can be said of Purpose #2. This purpose 
really demands a needs analysis expert, not a person 
skilled in evaluation. While the two may be similar, 
a needs analysis can be better done by someone trained 
in such procedures, rather than someone trained in eval- 
uation . 

Purpose #3 above is also not evaluation. Making pro- 
gram c planning decisions is decision making. If an en- 
terprise wants to hire someone to make decisions for them, 
to Improve their decision making, to insure that the enter- 
prise makes "good" or "the right" decisions, then the en- 
terprise should hire someone trained in decision making* 

The fourth purpose is the one being agreed upon by 
more and more evalu?«tion "experts." Bvaluacion }ias ais its 
primary purpose the collection of data to be used as feed- 
back to decision-makers in order to provide a basis for 
their decision making, not to make their decisions for 
them. It is more than assessing student achievement, more 
than measuring the percentage of achievement of an instruc- 
tional objective. Rather, evaluation should be the collec- 
tion of specific data about a given program or project which 
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the decision-makers of that project want or that the enter 
prise d^ems important and which will be used by those de* 
cision-makers for decision making regarding the strengths 
and weaknesses of their particular enterprise. 



RE V I EW : An Introduction to Educational livaluation 

(1) The first step before beginning an evaluation is 
to determine the purpose for conducting it. 

(2) If your (the decision-maker's) purpose is to have 
data for decision making , then you are in the same 
area as educational evaluation experts (Cronback, 
Cuba, Stuff lebeam, Fortune, Hutchinson, Worthen, 
Provus, and many others). 

(3) If your purpose is not to collect data for your 
decision making needs, but some other purpose, 
seek an expert in that area, e.g. public rela- 
tions. 

Having come up with an answer to "Why do J want to 
evaluate?" the next step is to consider some basic con- 
cepts of evaluation. 
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SOME BASIC CONCEPTS OF EVALUATION 

The term "evaluation" is an all-encompassing concept 
in education today. Many processes are termed "evaluation" 
when in fact they could probably better be called something 
else. Some examples will show how fuzzy a concept "evalua- 
tion" can be . 

The testing of products to describe their character- 
istics is called evaluation. Why not simply call it product 
tt6ting? The accumulation of data about an institution's 
/ operation, its income, expenditures, costs per credit hour, 

^ faculty- student ratio, etc,, is called evaluation. Why not 

_ simply call it institutional accounting? The measurement 
of pupils' knowledge at the beginning and end of a course 
is called evaluation. Why not simply call it achizvcmznt 
I testing? (Pace, 1968, pp. 1-2) 

^ These are a few examples which show some of the differ- 

ent, things called evaluation. Yet each of these is not 
evaluation. Evaluation is different. The purpose of this 
section is to discuss what is and what is not evaluation. 

Traditionally, evaluation has been conceived of as ad- 
ministering of a test, usually standardized, for the purpose 
of determining something, usually student achievement. Eval- 
uation has also been viet;ed as the way in which one deter- 
mines "how good" or "how bad" something is as compared to 
something else, i.e., Program A to Program B, or School A 
to School B. 

This approach can be labeled the T/iaditional Mode£ 
0|J iiatuation. It is usually implemented in the following 
manner: an outside expert (consultant) is hired to do an 
evaluation, he looks around for a few days to get a "feel" 
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for the enterprise, selects a set of standardized tests that 
fie thinks has something to do with the enterprise and ad- 
ministers them, both pre- and post-. The results, which 
often show no significant differences, are then written up 
in the form of a report. This report is then submitted to 
the decision-makers who may refute it, applaud it, but most 
certain will file it on the bookcase. Such reports used to 
help decision-makers seldom are used to make decisions about 
effectiveness, or change, etc. 

Some of the evaluation experts cited alone would con- 
tend that this approach does not meet the decision^making 
needs of educators. This is not a sound procedure for con- 
ducting an evaluation, e.g., simply pre- and post- testing . 
Although ''evaluation" and "testing" have usually been used 
interchangeably in educational research, evaluation is more 
than just testing. 

This conception Bvaluator-as-Expert-Model of 
evaluation is a narrow one. Also, it is usually not very 
useful to the decision-makers for whom it is done. In 
terms of the decision-makers involved, these types of eval- 
uations provide little, if any, useful data on which to 
make decisions regarding program strengths and weaknesses, 
redefinition and refining of program processes, etc. This 
is why so many seemingly excellent evaluations (excellent 
at least from the perspective of the researcher or the eval- 
uator) have been written, bound and then put on the shelf 
there to remain unopened and unread, with conclusions and 
recommendations ignored , not acted upon • 

The function of evaluation must be to provide this 
kind of relevant data to some decision-makers with respect 
to some project or program, i.e., data they will use for 
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decir^ion-making. Relevance here may be defined in practi- 
cal terms of data which the decision-maker sees as accept- 
able and useful to him and who will then use such data to 
help in making decisions* 

Another traditional approach to evaluation has been 
to have a Board of Experts come into an enterprise to do 
the "evaluation." This is found in its highest form in the 
Accreditation Model, with which most school personnel are 
familiar. The Accreditation Team looks at the physical 
plant, number of chairs, number of books, etc. It f^ccin't 
really look at program outcomes. Usually the Accreditation 
reports are very descriptive about very "physical" things. 
Quality of learning is not the focui?. 

Moving away from these traditional concepts of eval- 
uation, it is not only possible but essential to discuss 
a more effective and useful concept of evaluation. As 
Stuff lebeam has written: 

Evaluation is a science of relating ante- 
cedent conditions and processes to outcomes 
and outcomes to objectives. Evaluation 
strives (1) to determine the extent to 
which objectives are achieved - to meas- 
ure and define outcomes, and (2) to un- 
cover the functional relationships be- 
tween outcome and process variables - to 
explain outcomes. (1967a, p. 127) 

While this definition is not necessarily inconsistent 
with the pre- post-test approach, it does have to be 
viewed in conjunction with another concept; that of 
"decision-maker" and "decision making." 
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DECISION-MAKER AND DECISION-MAKING 

This concept is a relatively new one in the history 
of educational evaluation. In 1963 Cronback offered a new 
and somewhat more comprehensive definition. He defined 
evaluation broadly . . as the collection and use of in- 
formation to make decisions about an educational program" 
(Cronback, 1963, p. 672). This was the beginning of a new 
movement in the field of educational evaluation. 

^r^ce that rirticlc, cthor theorists and writers have 
taken up and expanded this notion producing most notably 
the Stufflebeam CIPP Model of Evaluation originated by 
Stuff lebeam and Cuba (1967a, 1967bp 1969). The defini- 
tion of evaluation is typified in the following: 

Project operations or activities are eval- 
uated to influence decisions which influence 
program operations which in turn evaluated^, 
ad infinitum (Cuba § Stufflebeam, 1968, p. 20). 

Stufflebeam also writes: 

. . . evaluation means the provision of in- 
formation through formal means, such as 
criteria, measurement and statistics, to 
provide rational bases for making judgments 
which are Inherent; in decision situations 
(1969, p. 53). 

These viewpoints sre represented in the literature 
dealing with the relatively new notion of educational eval 
uation as being decision-maker oriented. Taken together, 
they represent what could be called a Decision-Maker Model 
of Educational Evaluation. 

Another basic notion should be considered at this 
point: Who are the decision-makers? A decision-maker is 
that person or group of persons who are responsible for 
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making decisions regarding an educational enterprise. From 
the perspective of the evaluator, the decision-maker (s) is/ 
are the person(s) for whom data will be collected and to 
whom the collected data will be reported for the purpose of 
assisting or aiding the decision making efforts. 

In the Decision-Maker Model, often the project, or 
program or school personnel are the decision-makers and 
further, their role as decision-makers is legitimatized 
in this Model. That is, this approach to educational eval- 
uation assumes: 

1) That the project or enterprise decision-makers, 
be they classroom teachers, principals or the 
superintendent (all of whom are potential de- 
cision-makers) have the right -- both morally 
and ethically and legally to make their own 
decisions about their own enterprise. 

2) That it is the responsibility of the project or 
enterprise decision-makers to make their own 
decisions. It is neither the responsibility nor 
the role of an outside "expert" or ''consultant'* 
to do chis . 

3) That the only legitimate purpose of educational 
evaluation is to provide :r?formation to these 
decision-makers for their own use as they see fit. 

4) That the validity of this approach is determined 
in the final instance by whether and to how great 
a degree the data is used by the decision-makers 
in making their decisions. 

This approach is based on a number of other assumptions, 
many of which separate this approach from more traditional 
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ones. First, it assumes that decisions can be made more 
effectively with appropriate data, Iinplicit in this pur- 
pose is that data, to be appropriate, must come from the 
decision-makers' individual projects, not from some exter- 
nal sources; and furthermore, that the decision-makers in- 
volved must believe in and be ready to use the data that is 
to be collected. Thus, evaluation takes on a new relevant 
usability in decision making when based on internal needs, 
wants, criteria and data rather than on the "irrelevancy" 
of external (and therefore probably unrelated) standards 
and criteria when these are imposed on a project. 

This conception, also demands that the decision-makers 
involved have the final say in the determination of what 
data they want and need to make the kinds of decisions 
they deem important and necessary, not data defined solely 
by an evaluator, or data determined by arbitrary external 
criteria. In other words, decision-makers should be able 
to tell the evaluator what data they will want and need, 
rather than the evaluator telling the decision-makers 
what data they will need. 

It is assumed further that evaluation is not a one- 
shot, po6t hoc procedure, where if the tests show you have 
succeeded by 90% you can sit back and relax, patting your- 
self on the back (although not knowing where you succeeded 
and where there is still room for improvement). Or conver- 
sely, if the tests show you failed, e.g., achieving only 
201, yoLi groan and chalk up a lost year, still not knowing 
where you failed or what parts if any are working. To be 
effective, evaluation must be built into a program from the 
first so that the constant and continuing decisions which 
need to be made about a program can be made on the basis 
of data wherever and whenever possible, rather than on 
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impressions or intuition alone « 

Finally, evaluation demands that before any data are 
collected, the decision-makers involved need to know not 
only what data they want, but also what data they will need 
and will use, why they want it and how they are going to 
use it. In other words, they must define the goals of their 
project in order that appropriate data may be gathered. 
Notice here that this is an internal problem, not an ex- 
ternal one. 

An evaluator's job within this framework of evaluation 
is to assist the decision-maker (s) in stating project or 
enterprise goals, in deciding what data is to be collected 
and how it might be collected. An evaluator's job is not 
to dictate which goals are important, which goals should 
be chosen, what is ''good** or "bad" and so on. 

This approach to evaluation is essential to decision- 
makers who are concerned with how well they are doing by 
their own standards, where they are failing and so on. 
This approach does not tell the decision-makers vokat de- 
cisions to make, but rather only shows them u;he^e they 
need to be made. 



II. THE FIRST STEP IN EVALUATION 
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II. THE FIRST STEP IN EVALUATION 



At this point, some decision-maker in the enterprise 
makes the decision (and follows through on it) to evaluate 
or have an evaluation done. He contacts an evaluator and 
sets up an initial meeting. What kinds of things should be 
expected at that first meeting? What should the decision- 
maker look for? What should he ask and expect to be asked? 
This section of the booklet focuses on these questions. 



NEGOTIATION OF THE CONTRACT: INITIATION OF THE EVALUATION 

The purpose of this first meeting between the evalua- 
tor and the decision-maker who has been responsible for set 
ting up the meeting is to develop the scope of work for the 
evaluation.* What kind of decision-maker would organize 
such a meeting? It could be the assistant superintendent 
who has been asked by a group of teachers, or the superin- 
tendent or some other decision-maker (s) to contact an eval- 
uator. It might be a team leader or a principal who feels 
a need to have an evaluation done. In short, it could be 
any decision-maker who has some legal (and moral/ethical 
responsibility); or has financial ability to bring in an 
outside person to do work, in this case evaluation work. 



If in a particular situation the administrator and 
evaluator are the same individual, the administrator would 
still define the scope of work of the given evaluator. 
These same questions would need to be answered. 
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Assume now the evaluator hns come to a meeting with 
the project or school or enterprise personnel • A number of 
areas will be discussed. The decision-maker should expect 
to be asked the same question posed a few pages earlier: 
"Wfi{/ do you tjoant to (Lvatuatzf'^ The purpose of asking this 
question is to make sure that it really is an evaluation^ 
that it is needed and wanted and not something else. If 
the purpose is to provide some kind of data for decision 
making, then the majority of educational evaluators prac- 
ticing evaluation today will probably continue the dis- 
cussion. If some other purpose is given, then the eval- 
uator should try to help the decision-maker specifically 
define the purpose of the evaluation and then suggest an- 
other type of consultant who might better achieve that 
purpose (e.g., a PR man or a needs analyst). 

Following agreement on the purpose of evaluation, 
the next likely thing to happen is for the evaluator to 
begin to explain what he or she can and can't do in terms 
of an evaluation. The decision-maker at this point should 
look for what tasks will be accomplished, by whom, and so 
on. The decision-maker should feel free to ask any ques- 
tions that might be bothering him and to clear up any con- 
fusion he feels. 

If at this point the decision-mailer and the evalua- 
tor feel comfortable with their respective positions, then 
the discussion should get more specific. The decision- 
n^aker i^hould expect to be asked something like "W/ia>C 4.6 Jit 
that you voant eva-daaterf?" The evaluator might also be con- 
cerned with what the purpose of the enterprise is; how 

^Reference is made to the footnote on the previous 

page. 
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complex it is, i.e., are there many parts and decision- 
makers involved, or is the enterprise small enough to be 
viewed as a sinnl' project or program? If the evaluator 
feels that the ent .rprise is too broad or too vaguely de- 
fined, he will probably try to help the decision-maker 
narrow it down. 

/ example, an assistant superintendent har^ invited 
i : evaluator to an Initial meeting. He says: 
"I t/oant my 6ckoot 4{/4tem (Lvaluattd." 

The evaluator sees this description of the enterprise 
as somewhat broad and responds: 

"Voa (joant the, tjohott thing eva^aa-Ced?" 
The decision-maker responds: 

"We-d£, not thz vohott thing, bat the fitadtng p^ogKa^it." 

Again, to make sure this is the enterprise to be eval- 
uated, the evaluator might ask: 

"The vohole. n^zadtng pn,ogn,am, 6y6tzm-voldz?^* 

"Wot Ktalty, ju6t thU neu; n^tadtng cu^^itcalum u)z have 
in thz l\odtt Ettrmntan^y Schoot." 

In other words, the evaluator wants a fairly explicit 
description of the enterprise. He would probably go on to 
ask what are some of the major elements of the program; 
some of the major concerns, etc. He might ask for a brief 
description in writing. The decision-maker should expect 
such a discussion. 

This initial meeting will also deal with /leAoa^ceA. 
It takes resources to do an evaluation. Resources are de- 
fined as: staff time, secretarial and clerical support, 
duplication costs, decision-maker time, and money. In 
other words, people usually think of resources as a fancy 
name for "money" but money is only pan,t of resources. The 
decision-maker should expect to identify the resources 
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which will be made available to the evaluation effort. 
Again, this is going to probably be more than just quoting 
a dollar figure. If the evaiuator does not ask to have re- 
sources identified, then the decision-maker should raise 
such issues as: 

1. What {joiil w;e do and iohat voitt you do? 

1. Who l6 going to typz up and dl^tKlbutz pnogKZ66 
^^po^t6? 

3. Who \joltt pay ioK tho. phom catt6? 

4. Wh^^^ voltl mzzttng^ takz placz bLtiJO^zn tkz zvat- 
uatoK and tht 6ta{i{i xnvo^ued? 

5. Who \joltt oKganlzz and conv^m thziz fm^tlng^? 

6. Will thtKz be a ilnat KzpoKt p-itntzd {1^ appKo- 
pKlat^]? Who voltt do It? In hovo many dopl^i? 
Who voltt pKlnt data and collzct In^tKuimnt^? 

These are just a few of the kinds of issues that need 
to be resolved during this initial meeting with the evaiua- 
tor. If the evaiuator does not raise these issues, then 
the decision-maker had better, or he is liable to find a 
lot of hidden costs appearing later. Before the discussion 
concludes, then, the decision-maker and evaiuator should 
agree on a list of resources, including all those things 
mentioned above in addition to money. 

Another and perhaps more important issue which should 
be raised and resolved in this initial meeting (aid which 
is often overlooked in many evaluations) is to identify 
for whom the evaluation is to be done. An evaluation can- 
not be done outside of a particular context or in the ab- 
sence of specified people^ An evaluation is done for 
people who have particular needs for the information to be 
collected by the evaluation. (After all, the purpose of 
an evaluation is to provide information to 6ormom and for 
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that person, or group of people, to use this information 
for making decisions,) In other words > who are the decision- 
makers of this enterprise who will be provided with data? 
At first glance, this question may seem simple and obvious: 
"We££, I catted you Mk. Bvatuato^ to come fte/te 6o I am thz 
rfec^A^on-mafee/t. " Right? Not quitel The evaluator should 
respond with something like, "We£-t, do you mafee decX^-conA 
about thz p/iog/iam voz a^e going to eua^aate?" 

"Wttt, oi couA-^e." 
"A/te you tht only ontV^ 

^^No," the decision-maker responds, "thzKt afit thz 
teacfie'ti In tht pfiogfiam i^ho makz daily dtalhlonJb .^^ 
"U that allV 

"No, tfie p/ilnclpal al6o mafee^ Aome dzci^ion^ about it. 
Fo^ tkat mattz/i, 6o doz6 tht 6uptn,lnttndtnt. I(J you ^taxt 
to think about It, thtKt afit a lot 0(J ptoplz Mho make de- 
al^lon6 about oufi /izadlng pxogKam," 

As it turns out, for any educational enterprise, whether 
it is small as a single class or as large as all Title III 
projects in the country, there are many, many decision- 
makers, and not just those who are usually thought of as 
decision-makers (e.g*, administrators). For example, in 
an evaluation done of en experimental K-1, integrated 
day Title III project, the decision-makers identified in- 
clude: (1) the team teaching in the program; (2) the 
principal; (3) the other teachers in the school; (4) the 
superintendent; (5) the school committee; (6) the parents 
of the children enrolled in the program; and (7) the Title 
III office in Boston* Each of these different decision- 
makers wants and needs different decisions from the others. 
To collect different sets of data or information for each 
decision-maker in the above example would have cost a 
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^OKtunt. Each decision-maker would require a different eval- 
uation design. It is not only important to identify decision- 
makers ^ but one also needs to put them in some priority 
order. In all probability it will be impossible to pay to 
have an evaluation done for each^ evaluation will not be 
appropriate for all at the same time. 

Part of this discussion then should also provide for 
prioritizing decision-makers. There are any number of ^ays 
this can be accomplished but what is important is that it 
be made very clear to all parties at the initial evaluation 
meeting who will be getting information. 

A related topic is how much of the identified resour- 
ces will be allocated to each decision-maker. That is, of 
the total amount of resources, how much will go to the 
evaluation for the first priority decision-maker (s) , to 
the second and so on. For the example given of the experi- 
mental K-1 program mentioned above, 1001 of the resources 
were allocated to the highest priority decision-maker, the 
K-1 teaching team. It was decided, however, to report in- 
formation collected for them to the other decision-makers 
but not to do an evaluation for the others. Resources 
simply did not all^w for such a comprehensive approach. 

It should be noted that providing data collected for 
the primary decision-maker to other decision-makers in the 
enterprise does not constitute an evaluation of those 
''others." Such data may or may not be relevant to these 
others in decision-making. Thus, simply reporting data 
gathered for one specific decision-maker to other decision- 
makers within the enterprise is not "evaluation" for those 
other decision-makers. 

Remember, an evaluation cannot be all things to all 
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people. The scope and limitation of the evaluation need 
to be determined: What Jil to be dom? What volll be done? 
Fo/i 'horn will It be domf 

This is an o^'erview of what should happen at an initial 
meeting between a decision-maker and the evaluator,^ Again, 
any doubts a decision-maker has should be expressed and 
dealt with; any misunderstandings should be cleared up at 
this meeting; both the decision-maker and the evaluator 
should feel comfortable with each other and with what each 
wants to do and can do. 



*Even if the evaluator is not external to the enter- 
prise, but is someone internal to an enterprise, the same 
questions need to be raised and answered. This section has 
been written as though the evaluator is entirely separate 
from the enterprise, an "external" person. This point of 
view has been taken for illustrative purposes. There is no 
reason why an evaluator can't be an "internal" person. The 
mechanics of carrying out what has been described here may 
vary a little, but the basic aims and purposes presented 
here would be the same whether the evaluator is "external" 
or "internal." 
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REVIEW; Of the initial meeting between an evaluator and 
decision-maker 

(1) Has the purpose of the evaluation been discussed? 
Have all parties come to a mutual understanding? 

(2) Has the ''enterpr ise** to be evaluated been specifi- 
cally defined to the mutual understanding of all 
parties? 

(3) Have all questions been answered satisfactorily? 

(4) Has a list of resources which includes not only 
simply money, but staff time, secretarial support, 
materials, etc. (variables which affect "money") 
been identified? 

(5) Have the potential decision-makers of the enter- 
prise identified in #2 been identified? 

(6) Have those decision-makers who will receive eval- 
uation data been rank ordered, prioritized? 

(7) Have decisions been made as to what percentage of 
resources should be allocated to each decision- 
maker? 

(8) Has the scope of : ork and responsibility of the 
evaluator and decision-maker (or makers, if there 
are more than one) been clearly established? 

(9) Has the time period for the evaluation been clearly 
defined? 

Each question should be dealt with at the initial mee»:- 
ing between the evaluator and the decision-maker. This re- 
view section can be used to check or assess progress during 
the first meeting, and to determine what has or hasn't 
occurred. A decision-maker can then know to what he is 
commiting himself anc^ can act accordingly. 
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PREPARATION OF THE EVALUATION CONTRACT 

An actual contract or letter of agreement should be 
prepared following the initial meeting. It should include 
all of the information which was used to answer the preceding 
questions outlining purpose, enterprise, description of the 
allocated resources, decision-makers, time lines, and re- 
sponsibilities. In short, it should include all the topics 
which were agreed upon between the decision-maker (s) and 
the evaluator at the first meeting. 

Once the contract has been prepared, it should be re- 
viewed carefully by both parties. Both parties should 
agree, and be comfortable with, the points in the contract. 
Otherwise the contract should be changed. 

Two final points should be made here: 

(1) Unless the decision-maker is very satisfied with 
the contract and is happy with its provisions, it 
shouldn't be signed. Otherwise, there may later 
be cause for regret. 

(2) The decision-maker shouldn't accept the contract 
that simply says Mr. Evaluator and Model School 
agree to an evaluation for $X.XX. "Evaluation" 
is a fuzzy concept and car include (and exclude) 
many things. Before a contract is signed, a 
dec is ion -maker should know what he is getting 
and that he wants, needs, and will like what he 
gets. Be sure that the who, whi:!t, where, when, 
and how are clearly specified. Be sure that the 
responsibilities required of both the decision- 
maker(s) and the evaluator are also clearly 
specified. 
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III. THE PROCESS OF IDENTIFYING GOALS 



Whenever an evaluatioin is done, it should have as one 
of its steps some kind of goals process. The purpose of a 
goals process is to identify those intents, er aspiv/ations , 
or goals which the enterprise being evaluated is to accom- 
plish. The evaluation process includes data collection. 
What data should be collected? The answer to this ques- 
tion is: Data should be collected on those aspects of the 
enterprise that relate to the goals of the enterprise. 

The process of identifying goals of an enterprise is 
a very important part of evaluation. It provides for the 
selectiuii of variables as well as providing the basis for 
designing the entire evaluation. If the goals process is 
incorrectly applied, then data to be collected later will 
be less complete, less efficient and less focused than it 
should be. These three factors, in turn, will cause the 
evaluation to be less effective than it should be. In 
short, there can be no efficient evaluation without a sys- 
tematic, reliable process for identifying goals and putting 
these goals into some kind of priority order. 

Goals occur on all levels of specificity. They do not 
necessarily have the rigorous criteria of specificity at- 
tached to them which is prescribed for behavioral objectives 
by writers such as Popham and Baker (1970), or Mager (1962). 
Table I lists some of the possible differences between the 
two classes of phenomena. Goals embody the A^nttnti of the 
decision-maker and not just the verbalized, specific state- 
ment of what the decision-maker thinks his behavioral ob- 
jectives are or should be. 
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TABLE I 

SOME POSSIBLE DIFFERENCES BETWEEN GOALS AND 
BEHAVIORAL OBJECTIVES 



A GOAL - 



!• Is general, vague, not 
very specific. 

2. Is fuzzy; may oveilap 
with other goals; may 
be in conflict with 
other goals. 

3. Embodies real intents. 



4. Does not really com- 
municate specifics to 
others . 

5. May be stated in terms 
of anyboay, including 
inanimate objects . 



A BEHAVIORAL OBJECTIVE — 



Uses a specific behavioral 
verb . 

Uses a single specific 
verb object , excluding 
possibility of overlap. 



Reflects writer's ability 
to write behavioral ob- 
jectives. 

Communicates very well 
and specifically to 
others • 

Is stated in terms of the 
learner. 



Examples : 

1. To have individualized 
instruction . 



2 . Self - actual izat ion 



Autonomous learner 



4. Open classroom 
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The student must be able to 
correctly solve at least 7 
simple linear equations with- 
in a period of 30 minutes. 

Given a human skeleton, the 
student must be able to cor- 
rectly identify by labeling 
at least 40 of the following 
bones: there will be no pen- 
alty for guessing (list of 
bones inserted here) . 

The student must be able to 
spell correctly at least 80% 
of the words called out to 
him during an examination 
period. 

(These are taken from Mager, 
1962, pp. 45-50). 
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Goals versus Objectives 

!t is important to clarify terminology. For example, it 
is important to distinguish between the concepts of "goals" 
and "objectives." The word "goal" is used intentionally as 
opposed to a current catchword in education, "behavioral" or 
"instructional" objective. For there is a distinct difference 
between the "goal" concept and the "objective" concept: an 
objective is, or should be, a subset of the goal concept. 

Rather than asking the decision-maker to v/rite down all 
his behavioral objectives, a different approach is called 
for. This different approach is necessary for several rea- 
sons. First, the former approach assumes certain behaviors, 
skills and knowledges on the part of the decision-maker: 
(1) the ability to write behavioral objectives; (2) the 
ability to translate the decision-maker's purposes or in- 
tents into mzanjLngiul behavioral objectives; and (3) t:he 
ability to write objectives embodying all his intents. To 
assume these skills on the part of any decision-maker is 
>oth illogical and potentially damaging to the overall eval- 
uative effort. (For further discussion on this subject, 
refer to Hutchinson and Benedict, 1970; Benedict, 1970.) 

The decision-maker is asked what he would like his 
"enterprise" to accomplish, the word "enterprise" being 
defined as that entity about which data is to be collected. 
(An enterprise can be a school, project, class, program: 
that which is to be evaluated.) 

This approach, using an interactive relationship be- 
tween decision-maker and evaluator should yield an initial 
list of "goals." The most noticeable quality of this initial 
list is that these "goals" are usually vague or nebulous. 
Differentiated staffing, educate good citizens, graduate 

O 
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responsible Americans: all of these might be typical of 
the level of specificity of goals at this initial level. 
Even though they are stated as fuzzy concepts, they embody 
real intents and aspirations on the part of the decision- 
maker . 

It should be pointed out that fuzziness is not always 
"bad." It is "good" in the sense that it serves the pur- 
pose of allowing people to operate in the ordinary communi- 
cation process of the day-to-day world. Usually people 
communicate in fuzzy concepts, dream in terms of fuzzy 
concepts, and they aspire in terms of fuzzy concepts. If 
one proceeds directly to behavioral objectives, avoiding 
fuzzy concepts, there is great risk these behavioral objec- 
tives will not reflect the full scope of the goal envisioned 
by the decision -miker. 

What is important is that the list of goals elicited 
be as complete as possible, and may be expressed grammatic- 
ally in sentences, phrases or even words. Otherwise there 
is the possibility of missing or omitting what might be 
some of the most important intents of the decision-maker 
for the project. Beginning with fuzzy goals is possible 
because a methodology does exist for dealing with the 
"fuzziness" of goals: The Operational ization Process which 
will be discussed later* 

A goals process should have at least three major pro- 
visions: (1) a mechanism for generating a list of items 
or goal statements; (2) a mechanism for insuring the com- 
pleteness of the list; and finally, (3) a mechanism for 
ordering (or prioritizing) the list of goals. 



(1) Generating a list of goals : 

The evaluator should elicit the decision-maker's goals. 
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being very careful not to insert into the process his (i.e., 
the evaluator's) own goals, nor his ovm interpretation of 
the decision-maker's goals. Beware of the evaluatcr who 
debates a decision-maker's goals with him; who tells the 
decision-maker what he (the evaluator) thinks the decision- 
maker's goals should be. If the evaluator "forces'" a goal 
on the decision-maker which the latter really does not want 
or does not hold, then data collected on that goal will not, 
and cannot, be used for decision making. Here the evalua- 
tion either will be incomplete or will fail entirely, de- 
pending upon the extent to which this "forcing" has occurred. 

(2) Insuring completeness of the goals list : 

As was pointed out earlier, one of the purposes of a 
goals process is to arrive at as complete a list as possible 
of decision-maker intents. The test of completeness mechan- 
ism helps to achieve this purpose. 

Completeness in evaluation means, that within the re- 
sources available, all data which a decision*maker needs to 
make his decisions, are provided to him by the evaluation. 
To insure this, at each of many decision points throughout 
the evaluation, it is necessary to "test the completeness" 
of many different processes. By doing this throughout 
rather than at a terminal point in the evaluation, the de- 
sign becomes more complete; so the data provided to the 
decision-maker will also be more complete. 

The procedure for implementing a test of completeness 
works basically in this way. A decision-maker, when asked 
to think of a certain class or set of phenomena, may spend 
an hour or two doing just that. This thinking usually causes 
him to have a certain psychological "set" about those phen- 
omena; thus he becomes "locked" into a certain pattern of 
thinking. To ask him to keep thinking in this same pattern 
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is not useful for he has probably exhausted the process from 
that perspective, A test o£ completeness is meant to jolt 
him out of that set or pattern by offering or stimulating 
the decision-maker with a different perspective, a differ- 
ent set of phenomena. By reacting to a new set of goals 
from a different perspective, he would again have a certain 
psychological set. And, depending upon available resources 
at the various points of the evaluation, he would be pre- 
sented with yet another set of phenomena from a different 
perspective. It is very important, then, that the evaluation 
have some provision for insuring completeness of goals. 
Such tests of completeness --in the forms of other lists 
of goals should not be the evaluator's own goals but 
should come from within the decision-maker's enterprise. 

(3) Ordering the lif'it of goals ; 

Once the list of goals has been generated and tested 
for completeness, it is necessary to put it in some sort 
of order. This list may contain anywhere from one to one 
thousand goals. It is impossible physically (and finan- 
cially) to proceed with an evaluation on twenty or thirty 
fronts at the same time. Rather, it is necessary to pro- 
ceed at one point at a time. A "prioritization" mechanism 
provides for a systematic ordering the decisioR-mak^r ' s 
goals so that the evaluator will know how to proceed. It 
is very important that the decision-maker decide this order, 
but with the evaluator assisting him in an objective and 
systematic fashion. The evaluator should not determine the 
order for the decision-maker. 
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REVIEW: The Process of Identifying Goals 

When an evaluation is being done, does it: 

(1) Use the decision-maker's goals? 

(2) Ensure that the poals are really those held by 
the decision-maker? 

(3) Ensure that the evaluator does not interfere 
by inserting his own goals or feelings? 

(4) Identify as many of the decision-maker's goals 
as possible? 

(5) Include an ordering process of some kind which 
is acceptable to the decision-maker? 
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IV. A PARTS PROCESS 



Unless resources including time, staff and money 
are extremely limited, an evaluation design should have as 
one of its steps a "parts" process. But what is a parts 
process? 

One type of evaluation information which is often used 
could be diagrammed like this: 



PROJECT 



EVALUATION 



REPORT; SUCCESS 63% 



Sept. 



May June July 



Aug, or Sept. 



This evaluation was done near tne "end" of the project. 
We might term this a post hoc evaluation procedure where 
some sort of measurement or testing is done at the end of 
the project. This is a one-shot type of evaluation. 

But the next question is "So u/feat?" What usefulness 
is there in deciding that the enterprise is doing well or 
poorly or that it is 63% satisfactory? What decisions can 
decision-makers make on the basis of this? If the report 
shows 80% success in June, does the "project" congratulate 
itself? What if the report shows only 20% success? Does 
the project then chalk up a whole year to failure? Or 
does it ignore the report? Furthermore, what does 80% or 
20% successful mean anyway? 

In short, such information is of little use in knowing 

what succeeded or what failed. The utility of evaluation 

should be in knowing what paKt6 or domponantl or ttzmtnt^ 
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of the enterprise are working well and which are not working 
well. In addition, one needs to know this at the moment it 
is happening when there is time to make corrections or changes 
rather than after the project is ended. 

A decision-maker needs to be able to assess each part 
or component as it contributes or fails to contribute to the 
purposes (goals) of the enterprise. 

Instead of looking at the enterprise as a whole, 

ENTERPRISE 

look at the components or parts or subsystems of the enter- 
prise : 



ENTERPRISE 





Component 1 




Component 3 






Component 2 




Component 4 











If one can see the parts of the enterprise, one can then 
evaluate each part as it contributes to the goals of the 
enterprise. The purpose of a parts process, then, is to 
identify the parts of the enterprise as seen by the decision 
maker for whom data is to be collected. 

Again, this is in keeping with the concept of provid- 
ing continual data to a decision-maker so that he may make 
decisions at any time during the project. One needn't and 
shouldn*t wait until it is all over and then either shout 
or cry. 

To identify the parts of the enterprise, the evaluator 
should work with the decision-maker. 



33 



This is not as difficult as it may sound. Every ''sys- 
tem" jr part has a certain number of givens. These include 



Input : 



Output: 



Those things occurring before the enterprise 
begins, or those pre-requisites for the pro- 
gram. Examples in a school situation might 
be an existent budget, a physical plant, the 
staff, and so on. 

Those things which are not a direct part of 
the project but which impinge on the project 
and thiis influence it. Examples of inter- 
faces might include the School Committee, 
parents, PTA, the Legislature, and so on. 

That which results after the program or a part 
of it is ended. In a school, the output might 
be the student as he is at the end of the 
project year. 



For evaluation purposes, what the decision-maker has 
to do is conceptualize these systems in terms of his own 
enterprise. The decision-maker should list the major con- 
ceptual components or parts of the enterprise in response 
to questions like, "When you thlnh youn. zntzKpKl^z^ 
what aKe, lt6 majoK paKt6; -cn te.Km6 vohat paKt6 do you 
think oi ItV 

The evaluator should not tell the decision-maker what 
the parts or systems are. He may tell the decision-maker 
about Input, Intzfi^atth and Output as general categories, 
but the evaluator should not fill in the category content 
for the decision-maker. The evaluator also should avoid 
giving the decision-maker too many examples because the 
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evaluation desigi;» might end up with someone else's ideas. 
If this were to happen, the evaluation would begin to lose 
its efficiency. 

Several other points should be made here about a 
"parts*' process. Different dec ision-makers may view the 
6amt enterprise (or :5y5tem) in different ways. Example I 
(page 35) shows components of a high school from the per- 
spective of the Superintendent, one of the decision- 
makers in such an enterprise. Example II (page 35) shov;s 
the components of the 6amt high school from the perspec- 
tive of the School Board, another decision-maker in the 
same enterprise. These two examples show how a single 
enterprise can be viewed very differently by different 
decision-makers within it, A third example -- Example III - 
(page 36) is also provided, which shows the components of 
an Early Childhood Program from the perspective of the teach 
ing team, the primary decision-maker in this particular 
evaluation . 

In the three examples given, the enterprise has been 
broken down into its parts. Identifying the r^ajor parts of 
the enterprise can be considered as being the first level 
of breakdown. Each of the systems at the first level of 
breakdown are in themselves systems. As such^ they have 
input, output, and interfaces, and other subsystems. 

The next step in a parts process is to go to the sec- 
ond level of breakdown for each of the systems identified 
at the first level. 

For example, look at the system labeled "Climate" in 
example Ilia. (Page 36) Climate is the first level of 
breakdown from Example III. In this instance, when broken 
down one more level, i.e., the second level of breakdown, 
two subsystems were identified: "Physical Climate" and 
"Affective Climate." 
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A HIGH SCHOOL 



Resources 



Interfaces 
with Environment 



Defining Processes 



Evaluation 



Operating 
Processes 



Tone and 
Context 



Exan;ple I , First Order System: Superintendent ^ s 



A HIGH SCHOOL 



Input 
Process 



Students 



Faculty 



Staff 



Procrams 



Pro j ec t s 



Centers 



Process for 

Defining 

Programs 



Process for 

Defining 

Projects 



Process for 

Defining 

Roles 



Physical 
Plant 



Inanimate 
Resources 



Output I 
Proi e s s 
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Information 
Systeir^ 



Adminis- 
tration 



Governance 



Evaluat ion 



Interfaces with Environment 



Example II > First Order Systems : School Board * s 
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INTERFACES 



CLIMATE 



PROGRAM 



INPUT 



CfllLDREN 



PROCESSES 



INANIMATE 
RESOURCES 



OUTPUT 



STAFF 



PRIMARY SYSTEMS 



Example III : First Order Systems: Mark's Meadow K-1 Enterprise 



CLIMATE 




CHILDREN 


INANIMATE RESOURCES 






boys girls 






affective 




4'5 5's 






physical 




6's 7's 










sibl ings 






PROGRAM 




PROCESSES 


STAFF 


communication 




integrated day 




teachers 


soc ial deve lopment 




pupil flow 




aides 


emotional " 




teacher flow 




interns 


physical " 




ind i vidua 1 izat ion 




director 


cognitive 




time f lexibi 1 i ty 




spec i^i lists 


aesthetic 




homogeneous , cross* 










level grouping 




independent study 










other 



Example Ilia : Second Order Systems: Mark's Meadow K-1 Enterprise 
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An evaluation design should provide, then, for some 
kind of ''parts" process, from the perspective of the 
decision-makers for whom data are to be gathered. The 
parts process, like the goals process ^ should have at 
least three major provisions: 

(1) a mechanism for identifying (or generating) an 
initial list (or set) of parts, 

(2) a mechanism to insure that all the major parts 
lave been identified, and finally, 

(3) a mechanism for matching goals to parts since the 
original purpose of parts was to he able to eval- 
uate the enterprise in terms of its parts, vis-a-vis 
goals, not the whole enterprise. 

The purpose of the first mechanism and what it might 
look like are described in the beginning part of this sec- 
tion. In terms of the second mechanism, as with goals, 
the objective here is to complete a systems breakdown as 
far as possible. The more complete and specific the an- 
alysis of systems, the more specific and meaningful can 
data be related to specific parts of the project.* 



*For a more detailed discussion cf systems analysis 
in education and how it might be used in planning and bud- 
geting not necessarily evaluation per se the reader 
is referred, for example to: Hartley, H. Educational 
Planning - Programming - Budgeti ng: A Systems Approach . 
Englewood Cliffs, New Jersey, Prentice-Hall, Inc., 1968 . 
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Concluding Remarks : 

Do NOT be alarmed, or frustrated, or depressed and 
throw up your arms and say, J'tt ntvCK be able, to do at I 
thl6.^ You're not supposed to the evaluator is. This 
material is being presented here so that when ycu hire an 
evaluator, you will know the kinds of things to look for, 
what to expect, and the purpose of various processes. This 
material is also being presented here so you will have some 
criteria against which to measure, guage or "evaluate" 
the evaluator and the evaluation. 

This material, it is hoped, will allow you as a de- 
cision-maker to go into an evaluation with your eyes open. 
By knowing what to look for, you should be a little less 
anxious. Evaluation is meant to help you . If it doesn't, 
then the evaluation is not succeeding and needs to be 
improved . 
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REVIEW: A Parts Process 

Let's review this section to see what to look for in 
an evaluation: 

(1) Does the evaluation make provision for providing 
data in terms of th^^arts of the enterprise? 

(2) Do the parts come from the decision-maker and not 
from the evaluator? (They should.) 

(3) Are there mechanisms for generating a list of 
parts; for insuring the completeness of the 
parts list? 
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V. A MATCHING PROCESS FOR GOALS AND PARTS 



Once the goals have been identified in the goals 
process, and the parts identified in the parts process, 
there is a need for a process relating the goals and parts 
to each other. A prioritized list of goals shou .d have re- 
sulted from the goals process and a prioritized ].ist of 
parts should have resulted from the parts process. Now, 
these need to be matched to each other. This is done to 
increase the efficiency and usefulness of the data which 
is to be provided for decision*making. 

One way of doing this matching job is shown in the 
example on the next page. The enterprise in this particu- 
lar evaluation is a high school course in mathematics and 
the decision-maker is the teacher of the math class. The 
goals, listed in the left column, were the teacher* s goals 
for the enterprise and the parts on the top row were also 
the teacher's. 

Wherever an "X*' appears in a box, it indicates that 
the goal in the column is supposed to be accomplished, at 
least to a degree, by that part of the enterprise. Each 
and every goal should relate to at least one part, and each 
part should have at least one goal related to it. Using 
a diagram makes it possible to observe if there are goals 
for which no part has been identified and to fill these 
gaps. Such a diagram also makes it possible to see if 
there are parts without any seemingly useful function. 

The evaluation should not tell the decision-maker to 

make a particular decision or even that a decision is 

needed. The evaluation simply provides the data and points 

out any apparent discrepancies, leaving the decision-making 
up to the decision-maker. 
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REVIEW; Goals/Parts Matching 

(1) Does the evaluation have a provision for matching 
the goals of the enterprise to the parts of the 
enterprise? It should, 

(2) Does this matching process use identified goals 
and identified parts for a given decision-maker? 
Or does it use one decision^maker * s parts and 
another's goals? (The latter shouldn't happen,) 

(3) Is the matching process performed by the decision 
maker? (It should.) Or, is the matching being 
done by the evaluator? ( It shouldn' t , ) 
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AN OPERATIONALIZATION PROCESS 



VI. AN OPERATIONALIZATION PROCESS* 



Operationalizing is one of the more important pro- 
cesses within evaluation. It deals directly with the prob- 
lem of translating what a decision-maker wants to do into 
an observable or measurable state. It is also an area 
where some current evaluation models such as Stuff lebeam' s 
CIPP (Context, Input, Process, Product) Model, Provus' 
Discrepancy Model and the EPIC Model fall far short of an 
ideal and in fact, do not satisfactorily deal with this 
process . 

After all these years, there is still a dichotomous 
trend in education regarding behavioral objectives. On 
the one hand there is Mager (1962), Bloom (1956), Popham 
(1969) , and Popham and Baker (1970) representing a school 
of thought which would define objectives in terms of 
minute, behavioral changes that are measurable. Without 
these changes, educators will never know where they are 
going or where they have been. On the other hand, there 
is an increasing movement with spokesmen like Atkin (1963) , 
Ausabel (1967), Raths (1968) and Eisner (1969) which ques- 
tion the efficacy of this approach. These writers suggest 
that when one is forced to operate along behavioral lines, 
the essence of what education is really about may very 
well be lost. They also argue that the behavioral objec- 
tives approach is limited in its ability to deal with 



The majority of this section originally appeared in 
Hutchinson, T.E. and Benedict, L.G., ''The Operationaliza- 
tion of Fuzzy Concepts," University of Massachusetts, mimeo, 
September 1970. 
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things that are or should be of concern and importance to 
educators, like affective goals. Despite Popham's (1968) 
excellent refutation of this latter point of view, an 
uneasiness still remains about the efficacy and desira- 
bility of one or the other of these two seemingly opposite 
points of view. 

Actually, these two positions may not be a dichotomy. 
The problem may really be that our abilities of conceptual 
izing are still too immature to simultaneously handle the 
non-Behavioral versus the Behavioral points of view. 

Evaluators, educators, and in fact all human 
beings, have enormous difficulties in re- 
porting the sum and sweep of their objec- 
tives. We all have goals and we consciously 
and unconsciously give priority to some 
goals over others. But we have few re- 
liable ways to report them to others or 
even to reveal them to ourselves. (Stake 
and Denny, 1969, pp. 375-376) 

This is the crux of the matter. We all have goals but 
getting from goals to verbalized or explicit statements 
of what these goals mean, not only to others but to our- 
selves, is the real problem. 

For example, it is easy to state that ''The student 
shall solve 5 quadratic equations in 5 minutes without 
the use of any materials other than scrap paper and a 
pencil. It is easy to communicate this to others with 
full understanding, as it is an easy task to determine 
whether this objective is accomplished by the learner. 
However, this is not the case with a whole host of other 
kinds of goals: ''The student shall be self -actualizing. . ^ 
or ''The student shall value his self,*' and so on. These 
latter goals are difficult to communicate and understand 
and yet a legitimate argument can be made that these are 
just as important as is solving 5 quadratic equations. 
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Yet, vhile verbalizinR these humanistic or affective goals^ 
teachers, educators and objectives-writers have failed to 
deal effectively vith such goals. This is probably because 
their conceptualizing abilities have not been advanced 
enough nor comprehensive enough to do so. 

What then is the solution? Or is there one? Is it 
true that without Behavioral objectives we cannot progress 
anywhere? Is it true, as the non- Behaviorists state, 
that putting content or goals into Behavioral terms de- 
stroys that which is to be measured? 

A possible bridge fro.Ti the Behaviorist to the non- 
Behaviorist position, a possible solution to this dilemma, 
has been developed by Hutchinson (1969a, 1969b, Hutchinson 
and Benedict, 1970). The operat ionalization of fuzzy con- 
cepts might allow both the Behaviorists and their oppo- 
sition to feel not only comfortable with what they are 
doing, but with each other. They need not seem to be at 
opposite points any longer, nor mutually exclusive, since 
in reality it is contended they are expressing different 
points on a single continuum. 

Examine for a moment the beginning of this contro- 
versy. Why is it that objectives ever began? It could 
have started when evaluation or assessment of student 
achievement began. It came into focus with programmed 
learning with which Mager was concerned when he wrote h^s 
boolc. The problem actually had its basis in the need for 
measurement. And this is the point at which evaluators 
entered the scene. 

Evaluators and evaluations have had and continue to 
have a bad name. They are associated with anxiety on both 
the teachers' and students' parts. They have too often 
been part of the first school of thought mentioned earlier. 



ERIC 



48 



"Tttt me youx 6ptcifiic bthavioKat objtctlvth and thtn I 
Witt tvataatt," is typically attributed to an evaluator. 
As Stake and Denny write (1969) , 

An evaluator* s technical skill should help the 
educator convey his purposes, both those that 
quickly come to mind and those implicit in what 
he does. What are the present methods . . . 
Our methods now are crude, unstandardized and 
unvalidated. They should be more evocative, 
more sensitive than indicated by the bold re- 
quest. Please state your objectives in the 
following spac^. (p. 376) 

However, the above is not the only shortcoming of 
evaluators. A second is that the subjective approach to 
evaluation is all too common a practice today. In this 
method of evaluation, the evaluator enters the situation 
and ''feels" what is happening, or tries to sense some sort 
of global d.lmensions of what's happening, afSer which the 
evaluation is written. The problems with this approach 
are obvious. 

Yet a third dimension which contributes to the fear 
and anxiety associated with evaluations is thac the eval- 
uator will use outside, unknown or irrelevant criteria to 
evaluate "my school" or "my course" or "ME." That this 
point has been compromised is evidenced, for example, by 
such criteria for a Social Studies Evaluation, as pro- 
vided in the Natura) Study of Secondary School Evalua- 
tion's, Evatuatlvt CKltdJuia (1960) as: enrollment, number 
of selections, range of class size, class periods per 
week, room arrangement and so on. 

These problems with the current state of evaluations 
need not be the case. In fact, the whole nature of eval- 
uation, what it is and isn't, what it should and shouldn't 
do, is changing (Stake, 1967; Stufflebeam, 1969; Scriven, 
1967). Evaluation is headed for a new definition for which 
it is indeed time. 
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It is in this new movement of redefinition of the func- 
tion of evaluation, and in developing a inuch-needed method- 
ology of evaluation consistent with this movement that 
Hutchinson has devised a procedure he has entitled "The 
Operational ization of Fuzzy Concepts." An initial reaction 
to such a title is probably scepticism followed by ^Wha\, 
it It?" Upon investigating this procedure, one discoveri> 
an extremely wide range of possible applications. One such 
application is dealing with educational goals that are not 
easily translated into behavioral objectives. 

What is a Fuzzy Concept ? 

Fuzzy concepts are common. We all use them every day 
of our lives in communicating: peace, love, democracy, 
patriotism and civil liberties are just a few examples of 
some of the many, many fuzzies used frequently today, Be- 
cause each of us has a different perception of the same 
word, such as those above, or phrases like self -actual i zat ion , 
individualizing instruction and student-centered learning, 
there often arises misunderstanding, disagreement, tension 
and even conflict. Frequently one hears the point made 
thiat what is really at issue is a semantic problem, a 
communication gap. This is due in part to the use of 
fuzzy concepts. 

Fuzzy concepts can also be said to represent the 
dichotomy between instructional (behavioral) objectives 
and goals (non- instruct ional objectives). This very im- 
portant difference or differentiation between goal and 
objective should not be underemphasized , overlooked nor 
confused, A goal, for example, is an "end" in non- 
behaviorally defined terms, such as "The, student shall be 
S^li-actualA.zi.ng." An instructional or behavioral objective 
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on the other hand is an oprAatlo^.alZnd goal, e.g., ''The, 
/student ^halt Li^t In mitlng at tza^t 5 dlKtctty o6.se7tva6^e 
components o<J hl6 6e,ti'Concp.pt as he ptKctlvts it.'' 

The apparent gap between the two schools of thought 
on the object ives controversy , between "goal s" and "be- 
havioral objectives/' is due in part to the fact that in 
reality these represent two different points on a single 
continuum, not two different continua. As Stake and Denny 
wrote, all of us have goals. The issue in thi,': contro- 
versy is simply a lack of conceptualizing strategies;, an 
absence of a means to show that the gap is only an -appar- 
ent one, 

Hutchinson's technique, the operational ization of 
fuzzy concepts, may be the conceptual tool needed l:o re- 
solve the issue. Keeping in mind the definition of goals, 
this might be represented as shown in Figure I, 



behavioral statement 

Operat ionalizat ion behavivoral statement 

GOAL of behavioral statement 

Fuzzy Concepts behavioral stateiP\ent 

behavioral statement 



FIGURE I. 

When the operat ionalization technique ^s applied to a 
goal, the process will probably yield many behavioral 
statements (or objectives). It is important, therefore, 
not to dismiss goals, just as it is important not to dis- 
miss objectives. The premise here is still the use of 
objectives, or operat ional i zed goals. What is important 
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is the way or means by which teachers and other educational 
decision-makers are exposed and introduced to the logic and 
necessity of objectives, as well as the way in which eval- 
uators go about arriving at behavioral objectives. 

Please note: the best way to learn this technique is 
to experience it. In order to maximize this experience, 
the reader is asked to practice each step of the procedure 
as it is introduced and discussed. To simply read through 
this section trying to do each step will not be very effec- 
tive for the reader* 



The Operationalization of Fuzzy Concepts: A Methodology 

Step 1: The first step in this procedure is for you, 
the reader, to choose the fuzzy concept to 
be operationalized. Some examples are: 
peace , love , helping others , j ob satisfac- 
tion, self -fulfillment , etc. The reader 
should choose a fuzzy concept that he uses, 
or intends to use, rather than one which is 
not important or meaningful tc him. For pur- 
poses of this paper perhaps it would be 
easier if the concept "helping others" is 
used. Write the fuzzy concept, "helping 
others," on a piece of paper. 

Step 2: Create in your mind a hypothetical situation. 

This hypothetical situation will have a group 
of people in it, an environment, things, fur- 
niture, etc. It may be indoors or outdoors. 
Now, imagine that the fuzzy concept exists 
in this situation and is in the epitome, is 
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absolutely 100% present. Observe that situa- 
tion and all the things you see about it that 
indicate to you that your fuzzy concept is 
present in this situation* The hypothetical 
situations should be as complete and real as 
possible . For example , the hypothetical sit- 
uation in this case might be a classroom with 
chairs , tables , blackboard, etc . There is a 
teacher present, a group of students and so on. 
The teacher's behavior is the epitome of "help- 
ing others." List those things you can ob- 
serve in this situation that indicate to you 
that the fuzzy concept is present, that the 
teacher is "helping others." Some things 
might be: 

a. concerned with the student as an 
individual 

b. warm 

c. sincere 

d. considerate of students ' opinions , 
values , etc . 

e • smiles a lot 

f. provides a supportive climate 

g . provides success experiences for 
students 

h- provides experiences for students to 
reduce their anxiety 

i. provides experiences for students to 
define and reach their own goals 

ObA^iously there are many others. Possibly none 
of these would appear on your list of your 
concept of "helping others." Now, you should 
write your list down. Use this hypothetical 
situation completely, try to identify all the 
elements of "helping others." 



53 



Step 3: Now again construct a hypothetical situation 
and again with the environment and furniture, 
things, etc., a group of people and there is 
present in this situation the complete ab- 
sence of the fuzzy concept, e.g., absolutely 
no "helping others" present . What things do 
you see in this situation that indicate to you 
that your fuzzy concept is completely absent 
from this situation? Let's take again the 
same hypothetical situation as was set up in 
Step 2: a classroom, a teacher, a group of 
students, etc. This time, imagine that this 
teacher is directly opposite the ideal of 
helping others. List those things you can 
see in this situation which definitely indi- 
cate to you this teacher is not "helping 
others." Soirje examples might include: 

a . ignores students * opinions and values 

b . not aware of students as individuals 

c . egocentric 

d. selfish 

e. does not allow for individualization 

f . authoritar ian 

g . discourteous 

h. undermines students' feelings;, morale, 
etc . 

Obviously these are only a few possibilities. 
Again, maybe none of these will appear on 
your list or fit your conception of "helping 
others." Write down all those things in this 
situation that you observe that indicate to 
you the fuzzy concept is absent. Don't 
bother with the negative statement of the 
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positive elements listed in the previous step. 
Concentrate on identifying those aspects that 
were not already found. 

Step 4: After having gone through both the positive 
and negative hypothetical situations, the 
chance of easily finding more dimensions out 
of one's mind is not very great. So next we 
employed some strategies called tz6t^ o£ 
(iomptztzm6 6 . (First test of completeness) : 
Get someone else to go through the same steps 
as above with the same fuzzy concept. One 
then looks at the other person's list and 
considers item by item if the item should 
be on one's own list and if it should, add 
it to the list. If you decide the item is 
inappropriate, reject it; i.e., it does not 
fit your conception. Or a third possibility 
is that the other individual's item may make 
you think of one or more dimensions you have 
forgotten (recommended perhaps because you 
dislike their dimension). Ideally this test 
of completeness should be done with three or 
four other people. Write down the appro- 
priate dimensions which result from above. 

Step 5: ^Second test of completeness): Go back and 
recreate the hypothetical situations. Now, 
there were things that you saw in those 
hypothetical situations that you wrote down, 
i«e., your two lists. There were other 
things that you saw that you did not write 
down. Go back, look again at those things 
that you saw and did not write down, and 
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seriously consider the implications of these 
not being dimensions. 

To use an example out of the context of 
"helping others," consider fuzzy concept 
"job success." If a person were operational- 
izing "job success," one of the dimensions 
which he rejected in the first hypothetical 
situation might be money. Now the question 
should be asked, "What a^e th^ ImpZlcatloni 
iox AucccA^ -cn a job w/ie/te th^ job pKovlddh 
no monay at att?" Suddenly it becomes ob- 
vious that for almost everyone money must 
play some role however slight in job satis- 
faction. So the dimension money is added, 
but perhaps a qualified amount, e.g. $10,000. 

Step 6: (Third and last test of completeness): The 
tas'^ here is to construct some dimensions 
that have nothing to do with your fuzzy con- 
cept, in this case "helping others," and 
again, consider the implications of these 
dimensions for your concept. Try that and 
in fact, write fhem down. Start out by 
asking you7;self, "What ka^ nothA^ng to do 

volth (fuzzy concept)" and then, "Poe^ 

it KQ^atty mattt^?" 

The example of our teacher "helping others" 
provided us with a number of dimensions of 
this concept. Now, did you consider the 
teacher's family life? Relationship with 
his or her peers, the administration? Prob- 
ably not, but is it not possible that each 
of these could have serious implications on 
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that teacher's '"helping others." The pur- 
pose here is not in fact to find things that 
have nothing to do with your concept, hut 
rather to attack the problem frow a differ- 
ent perspective. 

A? you proceed through these steps, each one 
will be more difficult as the dimensions 
that compri'je your conceptualization of 
what you mean by your fuzzy concept become 
more and more complete anJ the number not 
identified become fewer and f ev/er , therefore 
harder to find . 

After one has gone through the 6 steps in 
sequence, it is reasonable to conclude that 
one has a fairly complete list of the parts 
of the concept at the first level o£ break- 
down. This product of this process, then, 
might be presented in Figure II. (Page 57) 

Now using our example of helping others, as a result 
of the first 4 steps, some 17 dimensions of "helping 
others" were arrived at. Thus on the first level of 
Figure II (page 57) there are 17 numbers. The next step 
in the process is: 

Step 7: For each item en your list, in this case 17 
perhaps added to as a result o£ the tests of 
completeness, the reader should ask himself, 
"Can I ob6e,^ve, that dlrmn^^-ion dl^e,ctly?" 
Something which can't be observed directly 
is defined as a fuzzy concept. Thus, for 
each item you decide if it is still fuzzy, 
and if it is, then you must repeat, in the 
O same order, the sequence of steps above, 
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In this particular example, none of the 17 items are 
directly observable and thus each must be further opera- 
tionalized at least another level. Obviously at this 
point it becomes clear that this can be a very lengthy 
process. In fact, a complete operationalization , espec- 
ially of a very fuzzy concept, can demand more time (and 
energy) than is available. Thus at this point in the 
process, another technique is used, namely prioritization . 

Since time is a resource and all resources exist in 
limited amounts, the reader must decide how much time he 
can allot to operationalization, depending on the reason 
he began the process. As an example, let's assume time 
(and perhaps money) is limited to a given amount and the 
operat ionalizer decides only items 1, 2, 12, and 14 can be 
operationalized. He repeats the process for each of these, 
including the important Step 7. Again, if an unmanageable 
number of dimensions are found each of which needs further 
operationalization, the prioritization at Level II may take 
place, as in Level I. 

For a very fuzzy concept, what usually happens is that 
very few items at the first level of breakdown will be 
directly observable. As the operationalization process 
is carried further, a larger percentage are found to be 
directly observable. 

Perhaps it would be appropriate here to use a less 
fuzzy concept, one which can be fully operationalized in 
several levels rather than a large number. A fuzzy con- 
cept for a college physical education teacher might be 
••competent weight lifter. At the first level of break- 
down, there are two dimensions: Olympic lifts and power 
lifts. Asking the question, "An.e. the,6Z rma^u^ahlz on, 
ob6ZH.vabtz dimctly/' the answer is •"no** and the process 
is continued. 
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At the second level of breakdown, 6 more compjnents 
are found, three from each of the first two: press, snatch, 
clean and jerk; and bench press, squat and dead lift. Fur- 
ther operationalizing "competent,** certain attributes are 
attached to these dimensions, thus the third level of 
breakdown. 

For a weight lifter with a body weight of 123 i/2 lbs. 
or less, 

press: 150 lbs. 

snatch: 150 lbs . 

clean and jerk: 200 lbs. 
bench press: 200 lbs. 

squat: 250 lbs. 

dead lift: 450 lbs. 

Each of these can be observed or measured by numerous 
methods and thus are no longer fuzzy. The lifts them- 
selves are operat ionalized by the current A> A.U. Weight - 
lifting Handbook . (See diagram on next page.) 

This was obviously a simplistic fuzzy concept with 
appeal to a limited audience. However, it exhibits how 
the process can and does work. 

This then has been a brief overview of the operation- 
alization of fuzzy concepts. It was introduced by two 
potential applications: first, as part of a new methodol- 
ogy and second, as a method of resolving the objectives 
controversy. 

An operationalizat ion process should do the following: 

1. Deal vith the most important goals of the decision- 
maker for whom the evaluation is to provide data. 

2. Take the most important goal and systematically 
break it down into behavioral, measurable dimensions 
or components. 
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FIGURE III 
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3. Once the most important goal nsis been broken down, 
it will deal with the second most important goal 
and so on. 

4. Once operationalized , a goal or intent will con- 
sist of a whole list of observable or measurable 
items as in the weightl if t ing example. 

5. These observable items should be prioritized by 
the decision-maker (with the .^' aluator's help if 
necessary) . 

6. liach item now becomes the behavioral item for 
which measurement for evaluation will be done. 
In other words, each item becomes the focus of 
developing a measurement technique which is then 
implemented and data collected. 

The results of operationalizat ion , then, form the 
basis for developing measurement techniques. This is the 
reason for the importance of the process. If the opera- 
tionalization does not work, then data collection will 
fall far short of an ideal at best and may even fail 
completely. 
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REV I EW : An Operationalization Process 

An evaluation should have some kind of operat ionaliza 
tion process. It may not look exactly like the one de- 
scribed herein. It may look entirely different. But , 
there has to be some soit of operationalization process. 
This is essential because of the need to break goals or 
intents into measurable, observable, behavioral statements 
Merely starting with "write behavioral objectives" omits 
much that is important in terms of what the decision-maker 
wants to accomplish. 



VII. MEASUREMENT FOR EVALUATION 
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VII. MEASUREMENT FOR EVALUATION 



Obviously one of the most important parts of evalua- 
tion is the collection of data. Data is collected using 
various observational techniques. The decision-maker for 

data is to be gathered and reported has a very impor- 
tant interest in the techniques which will be used to 
collect data. Therefore, he should be involved in the 
development and/or selection of such techniques. 

If the purpose of evaluation is to provide data for 
decision-niaking ; and if the data provided is to be used 
by the decision-maker; then any techniques used to collect 
data must be perceived as valid by the decision-maker or 
he will not use the data. 

For example, if an evaluator is hired and he proposes 
to use a standardized test his concern or company has de- 
signed, the decision-maker should carefully examine the 
test to see if it took^ to him, the decision-maker, as 
though the information it will collect will be useful, 
that he v^fill be able to use it. If the decision-maker 
feels that most of the information the instrument will 
collect r/ill be useless to him "I>t rma6uK^6 thing4^ I 
am not doZng^' then it should not be used. Rather, a 
tailor-made instrument ^ or technique, or procedure should 
be used. 

Most educators have had, at one time or another, a 
course in basic testing or in tests and measurements. Two 
concepts that most educators remember are ''Validity" and 
"Reliability." Probably no two measurement concepts have 
been as referred to, cr over referred to, in evaluation as 
these two. 
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What U Validity ? 

A technique valid if it accurately measures what 
it intends to measure. For example, using a ruler to meas 
ure the width of a room is a valid technique, A ruler 
measures what it is supposed to measure: distance. 

There are many kinds of validity construct, con- 
tent, predictive but one of the most important, in 
decision oriented evaluation, and the one most frequently 
overlooked in ''evaluation'' is " dtct6ton-maktK vatidtty ." 
Decision-maker validity simply means: do you, the decis:^c 
maker, think that the data collection device suggested by 
the evaluator will collect the data that you want collec- 
ted, that will be of use to you? In other words, do you, 
the decision-maker, perceive the instrument as being valid 
(measuring what you think it is supposed to measure)? If 
the answer to these questions is "i^e^/' then tae technique 
or instrument is said to have decision-maker validity; it 
is acceptable to the decision-maker. If you, the decision 
maker, are skeptical about an instrument or measurement 
technique or have doubts about itr ability to do what you 
want it to do, measure what you want it to measure, then 
the instrument or technique is said to lack decision-maker 
validity and should not be used. 

What l6 Rtllabltlty ? 

Does the technique perform c<)ns istent ly with time? 
For example, if we had a ruler which expanded several in- 
ches on a hot day or contracted several inches on a cold 
day, it would not be a reliable measurement technique be- 
cause it would not perform consistently each and every 
time it was used. A technique has to be reliable (consis- 
tent) or it should not be used. 
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An instrument can be completely reliable and very 
"valid" in the traditional testing sense and yet supply to 
the decision-maker completely irrelevant data, useless for 
decision-making. In the past, traditional tests, testers 
and evaluators have concentrated on "validity" (not decision- 
maker validity) and reliability to the exclusion of the 
decision-maker's needs, (This is only one reason why so 
many "traditional" evaluations have failed, i e., have sat 
on the shelf and collected dust.) 

In evaluation, when it comes to the measurement, the 
decision-maker should expect some interaction with the 
evaluator on the development and/or selection of a tech- 
nique. If the decision-maker loaves the choice of tech- 
nique to the evaluator, it is probable that the data col- 
lected will not be completely useful to the decision- 
maker, possibly entirely useless. There is the example 
of the outside evaluator hired to come in and evaluate a 
summer workshop whose purpose was to take pre-school, 
disadvantaged children and give them readiness activities 
in preparation for their entry into first grade. The 
evaluator arrived with two tests in hand, both cognitive 
achievement tests, administered them, wrote up a report 
showing a few significant differences, (mostly no signifi- 
cant differences) and sent the report to the decision- 
makers. The decision-makers reacted: ^^Hzlthzn, tzit 
mzatafizd lohat voe u)ZK^ doingi" "We w;e^e dzatA^ng voltk 
emotions and attltudzi and he (Mr. evaluator) tz^tzd 
cognJLtlvt devetopmznt. " 

In this example, both tests had been field tested, 
were valid in testing terms and reliable but did not have 
decision-maker validity. As a result, the decision-makers 
rejected the whole evaluation, fired the evaluator and de- 
cided to find an evaluator who could develop and provide 



ERIC 



68 



measurement techniques which could collect data about what 
they (the decision-makers) were aictually doing. 

In the first place, then, an observational technique 
must fit that which it is to irteasure. It must be developed 
or selected from existing techniques ""^'or a specific task: 
collecting data on a specific goal or intent which the 
decision-maker may hold for his enterprise. Prepackaged 
tests, and standardi2<i;d tests often fail to do this since 
they are usually on such a general level (in order to 
measure a wide range of things) that they miss collecting 
data on the specific needs of a specific decision-maker. 

Part of decision-maker validity is a determination, 
by the decision-maker to insure whether a technique seems 
to fit that which it is to measure. If an instrument is 
clearly going to measure cognitive development and the 
major concern of the decision-maker is psychomotor activ- 
ity or affective components of that cognitive development, 
then regardless of how valid or reliable measure of cog- 
nitive development, the instrument will fail in this 
instance. It would not have decision-maker validity. 

"8a>t," the decision-maker is going to say, "How; do 
^ (tae) knovo about valldlty?^^ Sometimes it is just a feel- 
ing, an intuitive distrust based on experience, as with 
the example just given. However, there are a number of 
criteria a decision-maker can use to determine whether an 
observational technique is useful, valid, and going to 
serve his needs. 
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Criteria to Assess Observational Technique s^ 

The decision-maker can ask himself: tkz ttcknlquz 

dlKtdt ob^zxvat/.on b^havioK on, l6 jit IndlKZct ob^ZKyd " 
Hon ?" Direct observation is always preferred to indirect 
because it gives a much better indication of what is really 
happening. For example, if the item to be measured is 
•'children fighting in the schools/' it would be better to 
collect information by direct observation counting the 
number of fights per day than to give a self-report 
questionnaire to all the behavior problems in school ask- 
ing them to write down the number of fights in which they 
have been involved. Students and non-students alike know 
how to "distort" answers on a written test to the direc- 
tion the question asker wants. They know they are not 
supposed to fight so they report "no fights" when in fact 
there may have been several. In such situations direct 
observation is always preferable to indirect. 

Is the technique obtrusive or unobtrusive ? An ob- 
trusive measurement technique is something which is not 
ordinary but which is introduced only for the "evaluation" 
so to speak. Obtrusive techniques share the same problem 
that indirect measurement had above: it interferes with 
what is being measured and may very possibly alter the 
persons or behaviors being observed. For example, if the 
item to be measured is "cheating" (the peeking kind) an 
obtrusive technique is to have two or three persons stand 



*The term "observational technique" is a very broad 
one and is meant to include any data collection or measure- 
ment procedure, instrument, etc. It is much broader than 
the term "measurement" for example and is meant to include 
measurement as one subset. 
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in the room to watch for peeking. An unobtrusive measure 
might be to have a one-way mirror and to stand behind it 
and count the number of peeks. Unobtrusive measures are 
preferred where possible to obtrusive ones. Perhaps the 
best example is the annual or semi-annual trip by an ad- 
ministrator to "evaluate^' the teachers. The administrator 
comes into a teacher's room with his checklist or pad of 
paper, sits glaringly or smilingly in the back of the room 
busily writing. The teacher's behavioT will automatically 
change for the duration of this "obtrusive" measure. 
Whether the change is for the better or worse is not the 
point: the point is, what is being observed is not what 
is usually happening because the obtrusive technique is 
interfering and interacting with that which is being 
measured. 

A third criteria which can be used in assessing meas- 
urement techniques is that of natun.atm66 . Is the obser- 
vational technique to be used under natural conditions or 
under unnatural conditions (e.g., test)? That administra- 
tor was observing his teacher under natural conditions 
her natural classroom environment but he violated one of 
the other criteria. Thus it is important to note that 
having just one of x:h3 criteria may noc be sufficient. 
In the case of the teacher, perhaps again, observing 
through a one-way mirror would have been natural. (Granted, 
very few schools have such devices: remember, this is 
only for illustrative purposes.) 

There are other examples of "unnatural" conditions 
which the decision-maker can be on the look out for in 
reacting to or assessing observational techniques: sim- 
ulations, models, lab situations, test-taking conditions. 
Each of these is unnatural to an extent and will therefore 
distort to an extent that which is being measured. 
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An jidtdt obhZKyjCLtJiondL ^ecfin^qae then will be re- 
liable and valid (especially decision-maker validity) and 
it will also fulfill three other criteria: directness, 
unobtrusiveness and naturalness. But, as with all ideals, 
it is very seldom met. Meeting all of these criteria will 
be both expensive (usually) and sometimes impossible. The 
ideal observational technique for determining certain be- 
haviors o£ teachers is probably an invisible man. This is 
obviously impossible although highly desirable in many 
circumstances . 

However, knowing what is ideal, the decision-maker 
can then know how far a given observational technique is 
from the ideal. He can use these ''criteria" of the ideal 
to measure observational techniques the evaluator presents 
or develops. It becomes very useful for a decision-maker 
to have a rough idea in his mind of what an ideal technique 
Height look like for any given item to be measured. 

These criteria also become very important in the realm 
of the affective domain, psychomotor domain and in the areas 
of attitudes and emotions. In the cognitive domain, there 
has to be strong reliance on paper and pencil tests re- 
membering that even this is far from the ideal, and far 
from satisfactory in the other areas listed. 
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REVIEW: Measurement for Evaluation 

(1) Have you, the decision-maker, been involved in 
the development or selection of observational 
techniques? 

(2) Do the observational techniques ha\''e your "de- 
cision-making validity"? (That is, do you feel 
the data collected by them can be used by you? 
Meet your needs?) 

(3) Have they been field tested and been shown to 
be reliable? 

(4) How dlfizct is each technique? 

(5) How anohtKu^l\)2. is each technique? 

(6) How natuxat is each technique? 

(7) How far from the ideal is each technique and is 
this so far that it loses decision-maker validity? 

Again, these can be used as criteria by the decision- 
maker to know what he is receiving or is not receiving in 
the way of measurement in evaluation. 

BEWARE: the evaluator who has one or two or even more 
pre-packaged tests which he plans to administer regardless 
of what you, the decision-maker have to say about it. Such 
tests will probably not provide you with useful or useable 
information and therefore should be regarded with skepticism 
unless it can be shown that these are the very best avail- 
able. (This can be partially answered by going through each 
of the above 7 questions with the evaluator and posing them 
to him. ) 
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VIII. DATA COLLECTION 



Once an observational technique has been agreed upon 
by both the decision-maker and by the evaluator, then that 
technique is implemented and data collection is begun. 

There are several criteria which the decision-maker 
should be aware of when assessing the process of implement- 
ing the technique. Granted , the evaluator (or a measure- 
ment consultant who might be hired) has expertise in im- 
plementing observational technique^^ but there are certain 
things a decision-maker can also look at which allow him 
to make some observations or decisions about the implementa- 
tion of these techniques. 

First, u?/ien does the evaluator plan to collect data 
using a given technique? If the evaluator has planned to 
use a technique only once, at or close to the end o£ the 
project, then the decision-maker should question the ad- 
visability of this. Data should be provided on more than 
a terminal or after-the-fact basis. The decision-maker 
should make some reference to his needs for data before 
accepting a suggestion to use a technique once, for example 
when the project is nearly over or the school year is 
nearly over. 

Hou) ojtzn should a technique be used? There is no 
exact or correct answer to this question. For example, 
the following is a goal which is held by a teaching team 
for their enterprise, i\ this case, a primary classroom: 



In the room. 
The observational 
formation on this 



many children's thi 
technique developed 
is simply: to rand 



ngs are displayed. 

for collecting in 
omly pick a time 
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during the week; send an observer into the classroom to 
count all thingF displayed which are children's things 
(not teacher's things). (Children's things included: art, 
papers, things brought from home to show to the class, etc.) 
It was decided to implement this technique for the first 
time in October of the year. 

Time I: In the classroom there were 12 children's 

things displayed (dravrings, sculpture, 
papers, etc.) 

The primary decision-maker (the team of 4 teachers) de- 
cided that this was really not sufficient to meet their 
intents for this goal and so they decided they would work 
at xiicreasing the accomplishment of this intent. In this 
case, the technique was used again a week later and this 
time , 

Time II: 35 things displayed 

The team decided that they had reached a satisfactory level 
on this and would now turn to other things. 

This does not mean that the technique was never used 
again. It would be used again to see if this level was 
dropping off, staying stable or increasing (each of which 
would indicate a different set of conditions necessitating 
a different kind of decision). 

Time III: (4 weeks later): 39 things displayed (all 
of which were different from the 35 things 
seen 4 weeks earlier). 

This confirmed the decision-makers' perceptions and intu- 
itions that this goal was more than satisfactorily being 
met. In this case the technique might not be used again 
for 2 months. 

O 
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But, what if at Time HI there has been only 10 or 15 
things displayed » all of which had been on display when ob- 
served 4 weeks earlier? This would probably have caused 
alarm and would have allowed the decision-makers to deal 
with this in any number of ways» i.e., with any number of 
decisions. Evaluation does not tell the decision-makers 
what decisions to make or what caused the conditions nec- 
essitating the decisions. Evaluation provides data to the 
decision -makers which they then use to make decisions or 
not, as the case may be. 

The staff would immediately take action to correct 
the situation, in this example, making changes in their 
program, etc. The technique would be used again very soon, 
perhaps 1 or 2 weeks later, to see if the decisions c:nd 
changes made raised the number of observed items. 

The point of this discussion is: he frequency of 
use of a technique depends upon the needs and decisions of 
the decision-makers. A decision-maker should then be wary 
of the ev&luator who wants to simply give a post- test. 
Suppose in the above example, a "post- test" were given in 
June and it was found that only 10 things were displayed. 
If school were out for the summer, it would be much too 
late to do anything and it might indicate that thi5 par- 
ticular goal had been inadequately met, or in fact it had 
not been met at all. If such a "test" had been done in 
January, with similar results, half the year has gone by 
with a situation existing which really needed change. It 
is important, therefore, not to rely on such rules of thumb 
as post-tests. Implementation of measurement techniques 
should reflect decision-maker needs and decisions made. 

(Note again that in the example given, direct, nat- 
ural and unobtrusive measurement was doae. A questionnaire 
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was not given to the teachers to ask them what they did. 
Observation was carried out to determine it,) 

It should also be remembered that the frequency of 
use of a technique will vary from technique to technique, 
as well as for the same technique. Therefore, the decision- 
maker should not expect all the techniques to be adminis- 
tered or implemented on the same time schedule or with the 
same frequency. This would not be efficient, or focused. 
Such a rigid pattern of collecting data would not yield the 
most effective information. The most effective informa- 
tion is that which is available when a decision-maker needs 
it, in the amount he needs, and where he needj it. Col- 
lecting all the range of information all the time as would 
happen if all techniques were used the same would not meet 
this definition of effective. In fact, such an approach 
to measurement is costly and a waste, both in time and 
energy and money. 

Exactly when and how often a technique is to be used 
is a flexible situation. The decision-maker who wants the 
most effective evaluation should expect a flexible sched- 
ule of collecting data and should raise questions if the 
evaluator wants to administer or implement techniques with 
the same frequency and in the same time pattern. 

Sampt4.nQ : Another criterion where the decision-maker 
should expect to interact with the evaluator is that of 
sampling. Sampling becomes a very important criterion 
when one reaches the stage of collecting data (implement- 
ing observational techniques). The evaluator should pre- 
sent any sampling plan or procedures to the decision-maker 
in order to determine whether the plan has decision-maker 
validity. The decision-maker should expect such an event 
to happen. 
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I'^hat <i6 sampling? Sampling is picking a number of 
subjects from a larger group of them. For example, if there 
are 1,000 students in a school and one wished to determine 
how many were boys and how many were girls (assuming we 
didn^t have this information) a sample might be taken all 
from the population (i.e., all 1,000 of them). This 
sample might be 10%. (It is cheaper to only deal with 
100 than 1,000 in terms of time, money, etc) On the 
basis of randomly choosing a sample of 100, we find 55 
girls and 45 boys. We might then, on the basis of chis, 
Z6t4.matz what the percentage of each sex is in the whole 
population, 55% to 45%. 

This is a simplistic example to show that from a 
smaller sample, it is possible to estimate something 
about the larger population. If a population of students, 
or subjects to be observed is large, then some sampling 
should be done in order to reduce cost. Observing all the 
subjects in population is often expensive. This expense 
might be wasteful because sampling (when done scientific- 
ally and Ck **fully) can yield the same information, or a 
good approximation of it, which a census of the whole 
population would yield. In the 1972 national elections., 
a Gallup poll of only 1,500 people was sufficient enough 
and representative enough to show what the whole voting 
population would do. In the sample approximately 61% 
said they would vote for Mr. Nixon. As the election 
turned out, this percentage in the sample was almost 
exactly the population as a whole voted. 

Sampling is done not only to save time and money and 
effort, but also when it is impossible to find out a piece 
of information from all the subjects in a population (as 
in the example of the election.) There are two criteria 
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within sampling which the decision-maker should look for: 
4.^ze and >iep/teAen^ai^vene4^ . 

I£ one were measuring a goal of fighting in a school 
of 600, one would probably want to look at more than 6 stu- 
dents. A sample size of 6 from a population of 600 will 
probably be quite inadequate. The size of the sample should 
be large enough that the decision-maker is willing to gen- 
eralize from the sample to the population. Would a decision- 
maker generalize about 600 stcidents from a sample size of 6? 
It is unlikely. 

On the oth^T hand, is it necessary to observe att 600 
students to get an estimate of the amount of fighting going 
on in the scho-)!? Again, it is unlikely. A sample of stu- 
dents or a sample of classrooms will probably yield data 
which is valid enough to generalize to the school. 

The sample size, therefore, should be large enough 
(or small enough) to maintain decision-maker validity with- 
out overspending resources. If the decision-maker feels 
that the data which will be gathered from the sample will 
reflect the actual level of goal attainment in the popula- 
tion as a whole, then the sample size is sufficient.* 



*Obviously, sound sampling procedures need to be em- 
ployed and the evaluator may have to present some discussion 
on sampling to the decision-maker. The point here is that 
if the decision-maker does not perceive the sampling proced- 
ure as valid to him, then chances are he will not use in- 
formation collected from it. Sampling theory and decision- 
maker validity need to be brought together at this stage of 
the evaluation. This does not mean that sound sampling 
should be abandoned. It simply means that it must be 
perceived as valid and useful by the decision-maker. 
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There are certain scientific principles governing 
sampling and it may be that decision-maker validity alone 
may not be "scientific" enough to justify certain general- 
izations. The decision-maker should expect the evaluator to 
explain 5;uch principles during a discussion on sampling. 
However, if having to apply too many principles jeopardizes 
decision-maker validity to the extent that the decision- 
maker feels data to be gathered will be useless to him, 
then the criteria of decision-maker "validity" has not been 
met and the decision-maker and evaluator need to discuss 
the problem. There is no sense in gathering data which nc 
one wil^> use in decision-making. 

The second criterion the decision-maker should consider 
is that of the representativeness of the sample. Going back 
to the example of fighting in the school, it may be that the 
size of the sami^le has decision-maker validity, but that the 
representativeness of where that sample is to be taken does 
not. Let's say that the size has been determined to be 60 
students. If the evaluator has designed a sampling plan 
whereby all these 60 students are freshmen, when the school 
has four grades, then this plan is clearly not representa- 
tive. If, however, the goal was held for only freshmen, 
then a sample of 60 freshmen would be very representative. 

If the sampling plan calls for selecting students 
from only social studies or only from industrial arts, when 
the goal is held for English also, then the plan is not 
representative. The decision-maker, then, should carefully 
judge whether the sample is going to be representative. If 
he feelr> it is not, he should discuss this with the eval- 
uator. 

In the final instance, it is the decision-maker who 
will use (iata for his decision-making. It is the decision- 
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maker who will have to generalize from data gathered from 
a sample to the whole population. To do this> he will have 
to carefully as5>ess the size and representativeness of the 
sample. 
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REVIEW: Collecting Data 



(1) Is each technique dealt with individually with 
respect to how often and when it will collect 

data? 

(2) Does the schedule for collecting data provide 
for flexibility such that this schedule can be 
changed (anywhere from more often to less often 
depending upon the nature of the data collected?) 

(3) Has the evaluator discussed the sample and sam- 
pling procedures with you to determine your 
decision-maker validity? 

(4) Are you satisfied that the sample to be selected 
is representative of the larger population? 

(5) Are you satisfied that the sample to be selected 
is large enough to generalize to the larger 
population? 
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IX. HAVING EVALUATION DATA REPORTED TO THE DECISION-MAKER 
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IX. HAVING EVALUATION DATA REPORTED TO THE DECISION-MAKER 



"^Wfien l6 tkz data KzpoKtzd?^^ This very important 
question is one which i,s usually not addressed directly in 
evaluation and yet it is a crucial problem to consider. 
In many evaluations which have been done, the data is col- 
lected at one point in time with an evaluator's analysis, 
supnary, synthesis and interpretation followed by a written 
report. All this is then delivered to the decision-maker 
quite often well after the need for evaluation data has 
passed, e.g., in August, three months after the project 
has ended at least for the summer, or in September, two 
months after the in-service workshop has been conducted. 

This problem of reporting data well after it is needed, 
is one of tJie reasons evaluation has received a bad name 
and one T'\^i%^y^^ ili^-'^. many people have criticized evaluation 
as beTVig less than useful. What must be done is to collect 
-^i to report data as it is needed, not in one lump sum at 
some terminal point in ai project or enterprise. In the 
previous section which discussed data collection, the 
point was made that in some cases there is a need to collect 
the same set of data several times, especially when changes 
have been made in order to more likely reach a goal. To 
wait until the end of that class year for data will mean 
that decision-makers cannot make needed changes and the 
purpose of evaluation immediately becomes weakened. If the 
data is not reported until the end of the year, for ex- 
ample, it cannot be used as a basis for deciding whether 
or not to make a change. It is quite likely, in the ex- 
ample of displaying children's things, that even the need 
for making a decision would not be revealed. 
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to be truly effective, then, data for decision-making 
needs to be reported as closely as possible to when it has 
been collected. Also, the evaluator should be ready to 
collect the same data again in a short period of time if 
necessary. Data collection has to be responsive to decision* 
mak^i: .needs • 

What l6 to be KtpoKtzd t Again, this might seem to be 
a question with a very obvious answer but when it is con- 
sidered carefully, it will be seen that it is really much 
more complex than is usually thought. 

"The data l6 ^epo^^Ced." This is the answer. But, 
what comprises the data? Data can be considered as the 
information gathered by the observational technique and 
it will probably have some numbers or figures or charts. 
This is what many evaluations report as data. It is really 
a narrow definition because there are many other things 
which should be reported in conjunction with these number 
"data" which become important in the decision-making 
process. 

A data report should include many things besides the 
numbers. It should contain the following: 

1 . The name of the decision-maker for whom this par - 
ticular data was collected * It has been pointed 
out many times that there are many decision-makei s 
in an enterprise. If the primary decision-maker 
for whom this data was collected is the chair- 
person of the mathematics department, then that 
information should appear on the data report. 
"lAn'^ thlt obvious?" one migJ^t ask. If it is, 
fine; if it is not, then it should be. The other 
decision-makers in the enterprise, e.g., the 
mathematics teachers or the assistant superintendent 
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for curriculum and instruction or the principals 
will probably, at one point or another, also be 
given a copy o£ the data and it as essential that 
these other decision-makers know for whom and from 
whose perspective the data was collected. (Differ- 
ent decision-makers need different kinds of data. 
If the principal receiveir the data of the chair- 
person and does not know whose data it is, he may 
find it does not meet his needs since it was col- 
lected from someone else. This is why labeling 
is important . ) 

The name of the goal and its importance (or prior - 
ity) to the particular decision-maker . Take, for 
example, the earlier discussion of the goal "having 
children's things displayed.** This intent was one 
of the operational components of the more general 
goal "to have an affective climate in the program.** 
(The **display*' intent was only one of many other 
items. The data report for this particular item 
then should include the fact that this was part of 
the larger goal and that this larger goal was the 
#1 goal this particular decision-maker (the staff 
of 4) held for that program.) 

The importance of the operational component . The 
reader might be thinking at this point, "But having 
chA.td/izn' 6 things dli^playzd doz6 not 4^em to me to 
be a vz^y 4.mpoKtant paKt a^l^ctivt cllmatz.^^ 
The data report should also contain then the im- 
portance of the operational component to the de- 
cision-maker for whom it is being collected. For 
example, in this case, the report might contain 
**this component of display was ranked as number 27 
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of the 70 components of the goal "affective climate 
This information then gives other decision-makers 
information for their decision-making needs. 

4. The name (and description if appropriate) of the 
o bservational technique used to colle ct the dat a. 

5. The date of the data collection (or dates if appro - 
priate) and the place , e.g., September 17, 22 and 
28 in Mr. Teacher's class and Miss Teacher's class. 

6. The actual data, presented in terms which the 
decision-maker for iihom it is being collected can 
use and understand. 



These six items are important items which should be 
part of a report on data. They are items which the decision 
maker should expect. Such information clarifies the report 
and makes the data (in many cases) more effective, both to 
the primary decision-maker and other decision-makers of the 
enterprise. 
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REVIEW: Data Reporting 

(1) Is the data reported when it is needed? In the 
amount needed? On the appropriate iteins needed? 

(2) Does the report include more than just a few 
numbers and statistics? 

(3) Specifically, does the report include: 

a. the name of the person (s) for whom this par- 
ticular set of data was collected? 

b. the name of the goal and the importance of 
the goal which this data it being collected 
to measure? 

c. the importance of this particular operational 
component to the larger goal? 

d. the name and description of the observational 
technique? 

e. the date, time and place of data collection? 

f. the data? 

(4) Is the data presented in an understandable^ fash- 
ion? Such that it can be used and understood by 
the decision^maker for whom it was collected? 

These are criteria a decision-maker should look for 
and expect in a report of evaluation data. 
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WHAT A REPORT OF DATA SHOULD NOT HAVE 

Just as there are things which a decision-maker should 
expect and look for in a report on data, there are also 
things he should not find in sucli a report. If he does 
find such things » he should be skeptical about them and 
question the evaluator about their inclusion. 

The decision-maker should not find within such a re- 
port, decisions made by the evaluator on the data. De* 
cisions about the data, interpretation about the data, sig- 
nificance of the data: these are properly made by the de- 
cision-maker. The evaluator should not write such things 
as "Tfce4e aKZ good, thz pKojzat should contlnaz doing...** 
Or, "T/te^e aA.e bad, thz pKojzat should cfiange what It l6 
doing and do thl6..." Such conclusions and recommendations 
are outside the proper realm of the evaluator. Such in- 
ferences are for the decision-maker to draw. 

The report should not contain evaluator biases in the 
form of passing his personal judgments about the data or 
the techniques or the observations. Such personal likes 
and dislikes of an evaluator are outside the scope of eval- 
uation. (If the decision-maker wishes to hire someone who 
will come in and make such statements, then he should do so. 
However, such activity should not be called evaluation, but 
judgment.*) 



^Several educators writing about evaluation take a dif- 
ferent view about judgment. These writers state that the 
evaluator should also "judge" the enterprise, or the goals 
of the enterprise. Such approaches to evaluators over the 
years have tended not to provide very good data for decision 
making because such judgment has been perceived as coming 
from '^outside" agents who don't "know what we are doing," 
etc. The reader is referred to Scriven (1967, 1971) and 
Stake (1967a) listed in the references for a discussion of 
evaluation and judgment. 
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The report should not contain information from the 
evaluator which tries to influence the program in one direc- 
tion or another » which tries to have specific or particular 
decisions made, or which lists the program adequacies or 
inadequacies. These are in the domain of the decision- 
maker's r sponsibility. Again, if a decision-maker wants 
to hire someone to come in and make decisions, or recommend 
decisions then he should hire someone to do so, but he 
should not call it evaluation. 

The report should also not contain a section entitled 
"Commendations" for the same reasons cited above. Many 
evaluation reports contain a list of things which are 
•'commended" for the only apparent reason that the evaluator 
liked them. Such activ^ities are outside the legitimate 
scope of the evaluation. 

The same can be said of a section in many evaluation 
reports entitled "Recommendations." Such sections should 
be deleted for these are the responsibilities of the decision 
maker. Everyone likes to be commended but many (if not most) 
decision-makers would argue with such "recommendations" which 
of necessity must ref:iect a shortcoming at least as seen by 
someone. A kindergarten teacher will not argue with those 
things she is commended for, but in at least one evaluation 
where the evaluator overstepped his bounds and included a 
section of recommendations, the teacher, who was the primary 
decision-maker for this particular evaluation, disputed each 
and every recommendation with such responses as, "He doesn't 
undzn,6tand klndt^gaKtzn ch>t£d^ert," "He ji^n^ t an expe/it in 
taAty thitdho^o optn c£a4-&/ioom, " 

"He Ktaommtnd6 4uah and 6uch, u)ktch i6 not at att a goat 
tfie p/iog/iam." 

When an evaluator moves into the realm of "recommenda- 
tions" and "commendations," he moves out of the proper realm 
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of evaluation and into the realm of decision-maker for an 
enterprise of which he is not in fact a legitimate decision- 
maker. A decision-maker should bewari? the evaluator who 
wants to, or does, get into this area of decision-making 
for it is precisely that, decision-making. Decision-making 
is not evaluation. Evaluation should serve decision-making 
and it can do this far better by not trying to co-opt 
decision-making but by providing data to proper and legit- 
imate decision-makers. 
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REV I EK : what a data report to the decision-maker should 
not have 



(1) Does the report have decisions (personal) of the 
evaluator? (It shouldn't.) 

(2) Does the report have the personal likes and dis- 
likes of the evaluator? (It shouldn't.) 

(3) Does the report contain recommendations of the 
evaluator about the program, its direction, con- 
tent, and so on? (It shouldn't.) 

(4) Does the report have a "Commendations" section 
and a "Recommendations" section written by the 
evaluator? (It shouldn't.) 
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X. REDESIGNING THE EVALUATION 



Redesigning the evaluation is an option which occurs 
only in certain circumstances. Ordinarily, the decision- 
maker would not expect redesign to be part of every eval- 
uation but the topic will be discussed here so that the 
decision-maker might know what a redesign should include 
and when it might be done* 

If the evaluation has been done properly to this point, 
with the interaction of decision-maker and evaluator, and 
if the evaluator has been carefully fulfilling his role and 
not confusing his role with that of a decision-maker, and 
if the decision-maker is fulfilling his role conscientiously, 
then there will probably be no need for a redesign section 
pe>t ^e . Each step of the process, if the reader will re- 
member, has a kind of redesign part to it. A step is not 
complete unless it has been satisfactorily agreed to by the 
decision-maker and evaluator. For example, during the 
process of stating goals, the decision-maker must decide 
on which goals to include and which to omit. He must also 
decide on a priority order (with the evaluator providing 
the evaluation expertise necessary to help the decision- 
maker). If these processes are gone through and the 
decision-maker says, "Wo, that i-A not the goaZ^ tilt I 
Keaily hold,'* or "Wo, that l6 not the. pKloKlty oKdtK o^ 
my goat6," then that particular section is recycled on the 
spot. This could be called a redesign of the goals process. 

The same thing is provided for in each process of the 
evaluation. At lea3t, it should be. If necessary, a sec- 
tion is recycled or redesigned as a section until it is 
satisfactory. (Again, this is not likely to be necessary 
if the decision-maker has been actively and conscientiously 
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involved in the evaluation design as he should have been,) 

What are some circumstances under which redesign of the 
entire evaluation might be needed? Redesign might occur if 
or when: 

1 . The progra m or project changes dramatically or 
drastically . For example, the decision-maker 
within the project may leave, resign, die or be 
promoted, in effect changing the person(s) with 
whom the evaluator has been working and for whom 
the evaluation has been designed. This would 
necessitate redesigning the evaluation, 

2 . The emphasis of the program chang es (i,e>i the 
goal s change ) , During the course of a project or 
enterprise, goals are very likely to change. If 
this occurs, then redesign is necessary in order 
to reflect a change in goals or in priority of 
goals. This will, in turn, necessitate different 
observational techniques being designed, differ- 
ent data being collected, etc. 

3. The enterprise experiences a "break" or "gap '' 
between one part of its operation and another . 
This might occur in a Title III project, for ex- 
ample, which has been funded for three years. At 
the end of the first year, a decision might be 
made, or decisions made, which in turn would nec- 
essitate changing the evaluation. These decisions 
could deal with personnel change^ program changes, 
financial changes, content changes, etc. 

4. The enterprise is a long-term one . An example of 
this might be any part of a school system, e.g., 
mathematics curriculum, English department and so on 
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In this instance, it is a sound idea to have an 
evaluation redesign stage built in. So many var- 
iables can change during the course j£ an enter- 
prise, especially a long-term one that it really 
is necessary to provide for redesigning evaluation. 

A conflict, misunderstanding, or some similar 
problem, occurs between the evaluator and decision - 
maker . This might happen for example if the two 
parties did not understand their purposes and func- 
tions during the first step of initiating evalua- 
tion and that misunderstanding did not become ap* 
parent until some time during the evaluation. vSuch 
misunderstandings could include or focus on: the 
purpose of evaluation, with one party wanting some- 
one to make decisions and the evaluator designing 
an evaluation to provide data to the enterprise 
decision-makers. Another example might be that in 
the initial phase of the evaluation, the wrong or 
incorrect decision-maker was identified. The 
decision-makor who actually makes the decisions 
was somehow not properly identified. This, in 
turn, would mean that the evaluation has been 
designed to provide data to the wrong person and 
thus a redesign would be necessitated. 

Interpersonal relations-personality problems . 
As with any endeavor, these kinds of problems can 
enter the picture and could cause changes to be 
made. For example, the evaluator might have a 
value conflict with the decision-maker causing 
the evaluator to desire to leave the project. On 
the other hand, the decision-maker may experience 
value conflicts or personality problems with the 



evaluator and might cause him to ask the evaluator 
to leave. {^A reminder might be made here that in 
preparing the contract, there should be stipulations 
allowing for thi;. to occur without penalties to 
either party. A termination clause should be in- 
cluded for the mutual benefit of both parties, should 
the example just given arise. The decision-maker 
dof^s not want to be saddled with a person who is 
coij jletely incompatible with the needs of the 
decision-maker. Conversely an evaluator cannot 
provide the most efficient evaluation design if he 
feels that there are incompatible differences b*^- 
tween himself and the decision-maker. 
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REVIEW: Redesigning the evaluation 

(1) Redesign may or may not be part of every eval- 
uation. 

(2) If redesign is necessary, it may be so for any 
number of reasons. It would be impossible to de- 
tail them all here. They are the same kinds of 
reasons which can cause problems in any educational 
enterprise . 

(3) If redesign is necessary, then it should follow 
the same guidelines provided herein for a good 
evaluation. 

(4) Finally, redesign is going to cost additional 
resources: especially time. The decision-maker 
should consider this before making the decision 
to have a redesign carried out. 

(5) In the final say, it is the decision-maker who 
decides to have the evaluation redesigned or not* 

Observation of the evaluation process by the decision- 
maker using these guidelines (provided throughout this book- 
let) may provide the basis on which to make the decision 
that a redesign is necessary. This could happen as soon as 
difficulty occurs in the evaluation process, rather than 
finding out during the last month of the evaluation that a 
redesign is needed. However, such a decision to redesign 
when difficulty arises can only happen if the decision- 
maker has been checking the process all along the way* It 
is suggested that the guidelines provided herein could serve 
as criteria to check during the evaluation process, not when 
it is done. 
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XI . EVALUATION OF EVALUATION 



Evaluating the evaluation is part of the evaluation • 
Yet very few evaluations which have been done have had pro- 
visions for evaluating themselves. In fact, most evalua- 
tions which have been done in the past usually terminate 
with a Final Report, when it is too late to systematically 
evaluate that Final Report. 

One very important thing which a decision-maker should 
^xpe(^t is to have some provisions made for an evaluation of 
the evaluation. As with all the other processes of evalua- 
tion which this booklet has discussed, the decision-maker 
must actively participate in this process. 

If an evaluation is accomplishing its purpose, that 
is, providing valid data to the decision-maker for his 
decision-making needs, then certain events are occurring 
and certain events are not occurring: 

1. Data provided to the decision-maker is actually 
used by him (her, them) in making decisions. 

2. The evaluation is efficient: All the data col- 
lected for a particular decision-:naker is used 
by him. To the extent that data is collected and 
provided and not used, the evaluation has not met 
its purpose. 

3. The evaluation is complete: Of the decisions made 
by a decision-maker relative to a particular pro- 
gram or enterprise, as many as possible are made 
with data provided by the evaluation. 

4. The evaluation is focused: If data cannot be pro- 
vided (because of lack of sufficient resources like 
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time and money) for all the decisions ^ then it 
should be provided for the most important decisions. 

These three criteria efficiency, completeness and 
focus can be applied by the decision-maker to the eval- 
uation in order for him to determine the extent to which 
the evaluation is meeting its purpose of providing data 
for decision-making. 

It is probably impossible that any evaluation will 
completely meet these criteria. There are many reasons 
for this. First, evaluation efforts may be initiated too 
late in the course of the program or project to enable the 
data collected to meet the criteria. An evaluation cannot 
fully meet the criteria if it is not begun until half-way 
through the project. 

Second, resources will probably never be sufficient 
to allow the evaluation to completely meet the criteria. 
It is probably impossible to collect att the data, needed 
by dtt the decision-makers of a project to meet atl their 
decision-making needs, because the cost of doing this would 
be prohibitive. This implies certain things then which the 
decision-maker should take into consideration in evaluating 
the evaluation. The decision-maker must be cognizant of 
the amount of resources committed to the evaluation because 
resources determine the scope of the evaluation. He must 
remember that not all the data can be provided to all the 
decision-makers for whom it might be desirable. This is 
the reason that during the course of the evaluation, the 
pKlma^y dzc,Z6lon-makz^6 are identified and prioritized so 
that those persons most needing information might get it. 
This is also why the most important goals of the primary 
decision-makers are identified so that they might get in- 
formation on their most important needs or goals. If during 
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the course of the evaluation even one of these was done in- 
correctly, the evaluation will become less efficient, less 
complete and less focused. 

One way a decision -maker might collect information for 
himself so that he might evaluate the evaluation in terms 
of his own needs is to keep a log of decisions made relative 
to the program evaluated. Ideally, evaluation and the plan- 
ning of the program occur at the same time, prior to the be- 
ginning of the program. If they are not or cannot be, the 
decision-maker should remember that this will affect the 
evaluation of evaluation. For those decisions, he should 
note their relative importance to him. Then, he should 
assess whether and how much data was provided to him for 
those important decisions, and was it provided when he 
needed it. In other words, apply the three criteria. 

What are some other things a decision-maker might 
consider in performing an evaluation of the evaluation? 
Evaluation should not interfere with the enterprisers 
accomplishing its goals (unless the goals are in conflict 
with one another and then this becomes not a problem or 
fault of the evaluator but a decision-making problem). In 
fact, evaluation should help an enterprise to accomplish 
its goals by having information systematically provided 
during the course of an enterprise, so that the decision- 
makers of the enterprise can use it in their decision- 
making. 
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REVIEW: E valuation of Evaluation 

(1) lu the evaluation providing data for your decision- 
making needs relative to the identified enterprise? 

(2) Given the scope and resources of the evaluation: 

* Is the evaluation efficient? 

* Is the evaluation complete? 

* Is the evaluation focused? 

(3) Are you keeping a log of decisions you make rel- 
ative to the identified resources in order to be 
able to assess points mentioned above? 

(4) Does the evaluation or evaluator interfere with 
you and your enterprise achieving its goals? 
(They shouldn't.) 

(5) Finally, a person using this guide can evaluate 
the evaluation in terms of ZtA parts, e.g., the 
contract phase, goals process, parts process, and 
so on, if he monitors the evaluation using the 
criteria provided in each section. This would 

be done in addition to keeping a log of decisions 
(in 3 above) . 
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XII. WHEN RESOURCES FOR THE EVALUATION ARE REALLY SMALL, 

WHAT DO YOU DO? 



This booklet has tried to present an introductory pic- 
ture of the complex tasi: of evaluation for an audience in- 
tended to be educational decision-makers , particularly ad- 
ministrators • However, the reader may have received the 
impression that "Wzlt, thl^ alt ilnt and good, bat 7 
havt vtKy itvo Kt6ouKct6 and J ja6t can* t buy all o^ thi6.'^ 

Resources will always limit the scope of the evaluation. 
Limited resources will have to limit the scope but do not 
have to exclude doing evaluation entirely. Limited resour- 
ces simply mean that the evaluation will have to be mere 
efficient and more focused than unlimited resources. 

The evaluation must in fact fit, from beginning to end, 
starting to deliver usable data within the resources that 
are actually available to do the job. Therefore resource 
allocation becomes a very important part of the evaluation. 
All the resources can't be spent on any one part of the 
evaluation, e.g., identifying goals, or doing a parts an- 
alysis. If resources are small, really small, then what is 
needed is as complete a parts process as possible w;ZtftXn 
ljimZt6 . 

Limited resources will mean probably dealing with only 
one (the most important or primary) decision-maker of an 
enterprise. It will mean not doing a lot of tests of com- 
pleteness in the goals process. Possibly, because of the 
focused nature of the evaluation (on a very specific and 
well-defined enterprise) the parts process will be elim- 
inated entirely. 
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Limited resourcer> will also mean not operationalizing 
all the goals as completely as possible. It will probably 
mean operationalizing just the most important goal of the 
most important decision^maker . Throughout the evaluation , 
there will be s!iort cuts and shortened forms of the process. 
However, the basic processes should still be in the evalua- 
tion, even if in shortened form. 

Even very limited resources will not mean that a 
decision-maker has to forego a systematic^ focused and 
useful evaluation. An evaluation is always shaped by the 
resources. Even when abundant or limitless resources are 
available, there is a need for a focusing of it. 

By having some guidelines to use, a decision-maker 
can be aware of the shortcuts and shortcomings of an eval- 
uation as well as the strong points and ad^t^antages of an 
evaluation. Because there are limited resources, does not 
mean that the decision-maker should reject evaluation. In 
the final instance, evaluation, or providing data for de- 
cision-makers, is meant to help the decision-maker, not 
hinder him. The suggestions provided herein are intended 
to aid the decision-maker in the evaluative process. 
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(As Used In This Text) 

BzhavZon.at Objzctivt t A statement o£ what you want someone 
(usually a learner) to accomplish, stated in vzKtf 
6pzc<^<c behavioral terms. 

Vata joK Vzclhlon-lhoikJinci t This is the statement of the 

purpose of educational evaluation, first set forth by 
Cronbach in 1963 and now widely held by the leading 
experts in the field, including Stuff lebeam, Hutchinson, 
Cuba, Worthen, Provus and so on. It means that evalua- 
tion should collect and provide data to educational 
decision-makers in order that they might make their 
decisions based on data rather than intuition or 
"feeling. " 

Pec^^^on-Mafee/^ z Any person who in some way makes a decision 
about a particular project, program, endeavor or enter- 
prise. For a school, examples would be: students, par- 
ents , teachers , admini strators , staff , school committee , 
etvC. 

EntZKpKJii z: That about which data is to be collected; that 
which is to be evaluated; can range from a single lec- 
ture to a v/hole program or project (e.g., Title I or III), 
to a school, to a national program. 

Evataatlon : The act of identifying, collecting, and report- 
ing data to decision-makers for their decision-making 
needs. 

Fazzi4 Conczpt : Anything which is not directly observable or 
measurable is a fuzzy concept; a goal which is nebulous, 
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vague, general, e.g., good citizen, autonomous learner, 
self actualization. 

Goat I A statement of intent or an aspiration, something you 
want to accomplish; usually stated in fuzzy terms. 

Mtthodotosy * A standardized, operationalized, systematic 
set of rules and procedures for accomplishing a de- 
fined purpose. 

Hodtt : A generalized, non-specific set of general rules-^of- 
thumb or guidelines for accomplishing a purpose; a set 
of non-operational, fuzzy procedures for doing something. 

Ob^tJivatA.onal Te.chni(tue : Something with which to collect 
data, not just limited to a ^"test." 

OpeKatZonatiz^ ? To take a fuzzy concept and systematically 
put it into its specific, concrete, observable, measur- 
able states. 

VKloKltlzt i To put a list of items in order of most impor- 
tant to least important or from first occurring in time 
to last occurring in time. 

ReAoa/tce^ ? A term referring to money, time, staff, mater- 
ials, space, expertise: those things which are needed 
to carry out an evaluation. 
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